Jump to content
Sign in to follow this  

Looking for Googlebot?

Recommended Posts

Looking For Googlebot


I. AWStats


1. Log onto Cpanel

2. Under 'Stats', click AwStats Stats

3. From the menu located on the left side, click Full List (under Robots/Spiders visitors)


This will tell you what spiders have looked at your site. This is very generic information (i.e. no specific pages, no specifics of the actual spider, etc.). Also note, 'Hits' does not refer to the number of pages. One page generally involves many 'hits' (each graphic for instance counts as a hit). The amount of bandwidth gives you an idea of how much was 'taken' but again, be careful. What if the spider visited the same page multiple times (which is quite frequent). Again, I warn that this information is very generic.


II. Raw Logs - the 'Real Deal'


1. Log onto CPanel

2. Under FTP Section, click on Manage Accounts


At the bottom of the page, you will find how to download your logs.


Once you have your logs viewable, search for 'googlebot/2.1' this will show you all the actual pages that Google visited. A sample would look like this: - - [03/Jun/2003:21:15:53 -0400]"GET /dried_flowers-whl.htm HTTP/1.0" 301 342 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

The IP address (64.68.*) indicates this was a fresh crawl.

You can see the date of the crawl (June 3)

The file crawled (dried_flowers-whl.htm)

Note: the 301 means that this particular page had a 301 redirect


Further comment:


Logs are reset at the end of each month, so the logs of today began on May 31.

Share this post

Link to post
Share on other sites
This topic is now closed to further replies.
Sign in to follow this