Jump to content


Photo

Looking for Googlebot?


  • This topic is locked This topic is locked
No replies to this topic

#1 SEO

SEO

    Immediate Family

  • Members
  • PipPipPipPip
  • 1,370 posts

Posted 05 June 2003 - 10:59 PM

Looking For Googlebot

I. AWStats

1. Log onto Cpanel
2. Under 'Stats', click AwStats Stats
3. From the menu located on the left side, click Full List (under Robots/Spiders visitors)

This will tell you what spiders have looked at your site. This is very generic information (i.e. no specific pages, no specifics of the actual spider, etc.). Also note, 'Hits' does not refer to the number of pages. One page generally involves many 'hits' (each graphic for instance counts as a hit). The amount of bandwidth gives you an idea of how much was 'taken' but again, be careful. What if the spider visited the same page multiple times (which is quite frequent). Again, I warn that this information is very generic.

II. Raw Logs - the 'Real Deal'

1. Log onto CPanel
2. Under FTP Section, click on Manage Accounts

At the bottom of the page, you will find how to download your logs.

Once you have your logs viewable, search for 'googlebot/2.1' this will show you all the actual pages that Google visited. A sample would look like this:

64.68.82.32 - - [03/Jun/2003:21:15:53 -0400]"GET /dried_flowers-whl.htm HTTP/1.0" 301 342 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

The IP address (64.68.*) indicates this was a fresh crawl.
You can see the date of the crawl (June 3)
The file crawled (dried_flowers-whl.htm)
Note: the 301 means that this particular page had a 301 redirect

Further comment:

Logs are reset at the end of each month, so the logs of today began on May 31.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users