# /robots.txt file for http://webcrawler.com/# mail webmaster@webcrawler.com for constructive criticismUser-agent: *Disallow: /_privateDisallow: /cgi-binDisallow: /cssDisallow: /jsDisallow: /danosDisallow: /GuidesDisallow: /Guides/GuidesDisallow: /Trial
Disallow: /Updates