Google's robots.txt tool

Without realizing it was new I spent some time with Google's robots.txt tool. All and all a very nice new addition to the Google suite. I also got around to getting the sitemap module for Drupal so now it might show something in their cache that is newer than last July.

There is also a good comment that discusses why Google is not looking to support the crawl-delay directive that some other search engines support.

Category: