Quote:
|
Originally Posted by Petey
Could you expand upon this robots.txt file?
--Petey
|
The long answer is here
http://www.robotstxt.org/wc/robots.html
The short answer is what Brandon showed us. If you want to keep specific sections of a site from being indexed, then you disallow them.
Good form dictates that you have the robots.txt file, but in reality it's not necessary.
BUT be aware, not all robots obey the directions listed in the file.
A easy way to learn about them is to go the any home page and add a /robots.txt to the url and you should see their robots.txt file
www.cnn.com/robots.txt