ANNOUNCE: A Standard for Robot Exclusion

Martijn Koster (
Sun, 03 Jul 1994 17:49:55 +0100

Some of you may remember a proposed standard for robot exclusion
I posted to this forum last year.

In short it is a method of guiding robots away from certain areas in a
Web server's URL space, by providing a simple text file on the server.
This is especially handy if you have large archives, CGI scripts with
massive URL subtrees, temporary information, or you simply don't want
to serve robots.

After some recent discussions on the Robots mailing list some changes
were proposed, and a new version of this document is now available
from <>. Most of the
robots in operation either use it already, or have promised support

-- Martijn
X-400: C=GB; A= ; P=Nexor; O=Nexor; S=koster; I=M
X-500: c=GB@o=NEXOR Ltd@cn=Martijn Koster