Anyone can figure out how often bots visit their site by sifting through their log files everyday and organizing each bot and checking on the timespan between crawling, etc.
But who wants to do that?
Patrick Altoft created the Crawl Rate Tracker for WordPress. It does the grunt work to 1) visually display the rate at which Google, Yahoo!, MSN, and Technorati crawl a site and 2) provide data about how often each page is crawled and how many links that page has pointing to it.
The crawling of my site seems consistent with other web sites using the plugin. Yahoo! seems to crawl websites more often than any of the other bot, especially in cases where there are lots of incoming links.
My most crawled page is “WordPress 2.5 Almost“. It isn’t a particularly interesting post and it only has one inbound link. Why then is it so popular with Yahoo!?
When I wrote that post I referenced the screencast that Matt Mullenweg created and linked to the post. WordPress placed the trackback reference on WordPress’ PageRank 7 blog post. I know PageRank and Yahoo! don’t go together, but at least from Google’s perspective that page is pretty important, so surely Yahoo! thinks so too. Yahoo! crawls that page, on average, every four hours. The page is from March, has little content, and wouldn’t really be useful to anyone now, but maybe I can take advantage of that.
Taking Advantage of Crawl Rate Tracker
So if I know which pages are crawled most often, and subsequently which pages seem to have the most authority (at least with a particular search engine) then it would make sense to put that data to use.
On my most crawled pages I can now go back and add links that point to less crawled pages or pages I think that need to receive more authority.
We also learn that referencing other popular blog posts can be a great way to attract bot attention (maybe not from Google) and have your site indexed more quickly and possible receive more authority in the process.
There could be many more features to this plugin to make it even better, but what Altoft created is already leaps and bounds beyond what we had before. Go checkout his plugin. I’m hoping his plugin will allow for me to have much more control over the authority of pages and link juice flowing between those pages.
2 replies on “Track the Crawl Rate of Spiders on Your Blog”
Do you think the search engines are more likely to crawl pages that have THIER name in them, such as “Google”, “Yahoo”, “MSN”, etc? I’ve noticed that Google tends to find “Blogger” blogs VERY fast.
“MagicBrad”
Brad, I would say that mentioning the search engine doesn’t really help at all, but the fact that Google crawls Blogger more quickly is simply because they own it.