Re: Byte ranges (actually robots)

Gavin Nicol (
Thu, 25 May 1995 16:01:38 +0500

>>Isn't there a standard permissions file that robots can fetch?
>Yes, but that just let's me tell them what to stay away from. The issue
>here is not that they should stay away, but that I should recognize them.

Fair enough. I'd also like to recognise them, and some kind of
convention in User-Agent or somesuch seems desirable.

Putting a robot on our server is a great way of stress-testing it ;-)