# $Id$ # robots.txt RFC: http://www.robotstxt.org/wc/norobots-rfc.html # Validated by http://www.searchengineworld.com/cgi-bin/robotcheck.cgi User-agent: * Crawl-delay: 1000 Disallow: /cgi-bin/ Disallow: /backup/ Disallow: /admin/ Disallow: /manage/ Disallow: /system/ Disallow: /rexian/ Disallow: /inc/ Disallow: /include/ Disallow: /webmaster/ Disallow: /data/