Google spider
 

I was checking out the Whos online page and i found something really cool. I like how it now shows whos visiting the site from external links, including the Google Spiders. We're being spidered!!!! :p

Re: Google spider
 

Re: Google spider
 

I believe Google sends out these programs called spiders that crawl from page to page using the links that appear on each page, and store the pages on Google's server. They also count the number of links pointing to a particular page and that's how pages are ranked when you search for something.

Re: Google spider
 

Most search engines use spiders. You can read more about Google's spider here: http://www.google.com/bot.html

Re: Google spider
 

speaking of spiders..

look at the very bottom of the page. There is an 'Archive' link. That is our search-engine-spider-friendly archive that will get each & every thread in chiefdelphi.com onto google.com and the like.

Re: Google spider
 

I thought the robots.txt blocked out google's spider from spidering the forums. *shrug*

shameless plug --> http://www.mehh.net:8080

:D (Google hates non-standard ports and won't pick up my site :()

Re: Google spider
 

when i was looking at the 'who's online' page, i noticed something unusual. a "inktomi" spider was apperently registering on the forums. any idea how it is/was doing this?

Re: Google spider
 

I conducted some experiments while not logged in, and it looks like you are shown as "Registering" even when you are only just visiting the page that has the button leading to the registration form. So the spider probably was just following the link that appears at the top of every page.

Re: Google spider
 

Inktomi is a company that develops a web index similar to that of google. They have been becoming more active and are owned by Yahoo!. Yahoo! has plans to change from a google powered search to Inktomi, which would further thrust inktomi into the realm of (what my friend once called) "Really-Friggen-HUGE search engines. The spider that you see is Inktomi indexing the CD pages.

Re: Google spider
 

On a related note, I was just at the Who's Online page, and 9 of the 15 Guests where Inktomi spiders.

Wetzel

Re: Google spider
 

thats nothin'!!
on thursday, 85 of 98 guests (by my math) were inktomi spiders

Re: Google spider
 

Re: Google spider
 

Almost -- It means a spider would've already been to the site and let Google Search engine know that OCCRA is on the URL at ChiefDelphi.com. The Spider is a program that goes and looks at every link it can, and stores all the info needed to search for that page.