User-agent: * Disallow: /cgi-bin Disallow: /Pages/Bugs Disallow: /Archives/ Disallow: /Old/ # Google understands some extensions to robots.txt # http://www.google.com/webmasters/faq.html#nocrawl User-agent: Googlebot Disallow: /*.eperl$