User-agent: * Allow: / Disallow: /wp-a Disallow: /blog/tag/ Disallow: /download-media-kit? Disallow: /news/2007/ Disallow: /drupal Disallow: /custom-publishing Disallow: /files-corp/ Disallow: /js # # Robots.txt syntax is prefix-matching. /files-corp/ will do, # unless there are other files in /files-corp/* that needs to be crawled # Disallow: /files-corp/rates # Disallow: /files-corp/HWBA%2009-10%20factsheet(final).doc #