Re: I, Robot

Frank Atkinson (
Mon, 11 Dec 1995 17:46:21 -0500 (EST)

>From: Steve <>

>I may have missed it in the archives, but I didn't see any topics on which
>sections of a document a robot or spider or other Web traveller might examine
>to determine content information.
>I suppose that this sort of information could be contained in the <META> tag,
><META HTTP-EQUIV="Keywords" CONTENT="Audio/Video Equipment">
><META HTTP-EQUIV="Keywords" CONTENT="Laserdiscs sold here!">
>but is this indeed what robots look for besides (I guess) the first few lines
>of the document itself?
>If there isn't a specific indexing method that is standard for all search
>engines/robots/crawlers, shouldn't there be one?

There is a mailing list at (put subscribe in the
subject to subscribe) examining some of these issues as they relate to bots
which will go "shopping" on the web. There probably is a practical limit to the
amount of info which can be transferred along with the page before it slows the
page down. There is also an issue of human interface (which a web page is
designed to be) and a bot interface (which is designed to transfer info and
doesn't care about humans). Maybe a standard tag to a file or page with the bot
information would work.

! Frank Atkinson, DP Manager ! 614-228-3113 Voice !
! Hannah News Service ( Rotunda Inc. ) ! 614-228-5897 Fax !
! 16 E. Broad St. ! !
! Columbus, Ohio 43215 ! fratkins@freenet.columbus.oh !
! !
! The next generation !
! in shopping - AGENTS !