Re: Byte ranges (actually robots)

Gavin Nicol (gtn@ebt.com)
Thu, 25 May 1995 09:59:25 +0500

Messages sorted by: [ date ][ thread ][ subject ][ author ]
Next message: Steve Nisbet: "WEB Info"
Previous message: Kee Hinckley: "Re: Byte ranges (actually robots)"

>So what to do? If we recognize the user-agent as a known robot, we give it
>a flat presentation of the structure. The catch of course, is that there
>is no standard way to recognize a robot. Some convention in the UserAgent
>field would be sufficient, but....

Isn't there a standard permissions file that robots can fetch?

Next message: Steve Nisbet: "WEB Info"
Previous message: Kee Hinckley: "Re: Byte ranges (actually robots)"