Trailfire: All about robots

http://googlewebmastercentral.blogspot.com/2007/03/all-about-robots.html

sidisinsane's comments

Search engine robots, including our very own Googlebot, are incredibly polite. They work hard to respect your every wish regarding what pages they should and should not crawl. How can they tell the difference? You have to tell them, and you have to speak their language, which is an industry standard called the Robots Exclusion Protocol.

Other comments

There are no comments on this mark.

Similar Marks

View a Web Page as 'Googlebot' (by RiDdLeR)

Requesting removal of content from our index (by sidisinsane)

Official Google Webmaster Central Blog: The number of pages Goog... (by sidisinsane)

Google, duplicate content caused by URL parameters, and you (by sidisinsane)

Best uses of Flash from the Official Google Webmaster Central Blog (by Marje)

Webmaster Help Center - What is a Sitemap file and why should I have one? (by Marje)

Google Webmaster Central (by virginia7608)

Google Webmaster Central (by aardael)

Best uses of Flash (by sidisinsane)