I was checking out the Whos online page and i found something really cool. I like how it now shows whos visiting the site from external links, including the Google Spiders. We’re being spidered!!! 
Anyone want to explain what a google spyder is? I can’t seem to find anything online about it.
I believe Google sends out these programs called spiders that crawl from page to page using the links that appear on each page, and store the pages on Google’s server. They also count the number of links pointing to a particular page and that’s how pages are ranked when you search for something.
Most search engines use spiders. You can read more about Google’s spider here: http://www.google.com/bot.html
speaking of spiders…
look at the very bottom of the page. There is an ‘Archive’ link. That is our search-engine-spider-friendly archive that will get each & every thread in chiefdelphi.com onto google.com and the like.
I thought the robots.txt blocked out google’s spider from spidering the forums. shrug
shameless plug --> http://www.mehh.net:8080
(Google hates non-standard ports and won’t pick up my site :()
when i was looking at the ‘who’s online’ page, i noticed something unusual. a “inktomi” spider was apperently registering on the forums. any idea how it is/was doing this?
I conducted some experiments while not logged in, and it looks like you are shown as “Registering” even when you are only just visiting the page that has the button leading to the registration form. So the spider probably was just following the link that appears at the top of every page.
Inktomi is a company that develops a web index similar to that of google. They have been becoming more active and are owned by Yahoo!. Yahoo! has plans to change from a google powered search to Inktomi, which would further thrust inktomi into the realm of (what my friend once called) "Really-Friggen-HUGE search engines. The spider that you see is Inktomi indexing the CD pages.
On a related note, I was just at the Who’s Online page, and 9 of the 15 Guests where Inktomi spiders.
Wetzel
thats nothin’!!
on thursday, 85 of 98 guests (by my math) were inktomi spiders
[RIGHT]**After reading everything I’m still confused on what a spider is.
Is this right?
If I searched OCCRA on google and it gave me a CD link and I went to it I would be a spider on the forum?[/RIGHT]**
Almost – It means a spider would’ve already been to the site and let Google Search engine know that OCCRA is on the URL at ChiefDelphi.com. The Spider is a program that goes and looks at every link it can, and stores all the info needed to search for that page.