Googlebot Is Eating My Bandwidth

[Kathy_P]

After getting a notice that I was at 80% bandwidth I decided to look at Awstats. Turns out my not-viewed traffic for the month is almost double my viewed traffic, with Googlebot taking by far the biggest bite at 438.76 MB. For comparison, in Nov Googlebot used 165.96 MB of bandwidth and in Oct. it was 67.66 MB. What is going on here? I get the idea I need to look at my raw log files. I clicked on Raw Log File listed under Web/Ftp stats in Cpanel, but the file is .gz which my Windows XP doesn't know how to open.

1. Am I on the right track? If so,
   
2. How do I open the file? If not,
   
3. What should I be doing instead?
   

Any other suggestions or ideas would be appreciated.

[TCH-Rob]

Right click the file and select open with. Then you can select Notepad.

[Kathy_P]

As best as I can tell, a *.gz is some sort of compressed file. When I said Windows XP couldn't open it, I meant it didn't know how to un-compress it. Once it is uncompressed I am sure I will be able to read it. But I'm still not sure if it is the raw log for the month or just for today. It won't do me much good unless it is the log for the month.

[Deverill]

gz is a file compressed with the gzip format. There are programs that will read it that you can get for free from sites like download.com for instance. I use WinRAR which reads gz files.

 

Googlebot is a touchy friend/enemy. If you block it then they will think your site is gone and eventually remove your listing - not good. If you leave it alone it may eat your bandwidth - also not good. You could try a robots.txt file to restrict it from certain directories but depending on your site that may also not be good.

[TCH-Bruce]

Winzip will also open .gz files.

[TCH-Rob]

If you use WinZip it will extract a .com file I believe. That file can be read using a text editor. Sorry I didn't specify earlier, wasn't paying attention to the entire question.

[Kathy_P]

Yes, it did extract a .com file which was confusing to me, but when I told my text editor to open it, it could. I have to say after looking through the first couple thousand lines using the Find command, I really didn't know anymore than I knew looking at AWstats. I also tried a program called WebLog Expert, which allowed me to filter out everything except Googlebot, which allowed me to see that there wasn't any pattern to the visits. So I actually went to Google itself and read the FAQs here: http://www.google.com/webmasters/bot.html . And since my question wasn't answered I filled out the contact form. If I actually hear from them I'll report back.

[robok]

Hi TCH guru's....

 

Kathy is my sister and I have been trying to help her troubleshoot. I am the account holder of the reseller account and she is one of my customers.

 

It seems the bandwidth "of record", i.e. that which tells her she is "over her limit" is what I see in my resellers panel, but in her AWstats it is a lower number. How do these two relate?

 

For example, in WHM her bandwith used is 1.04 Gig but in her AWstats it shows 415 MB. Does the AWstats exclude some traffic?

 

EDIT: let me correct that.... I noticed there is Traffic Viewed and Traffic Not Viewed in AWstats, that seems to be the difference. Do you have any additional information about "Traffic Not Viewed"? That is nearly double her "Traffic Viewed"

 

Thanks,

Rob.

Edited December 28, 2005 by robok

[TCH-Andy]

Hi Rob,

 

AWstats only records HTTP access - ie. It doesn't include FTP, emails etc. The bandwidth used should be the same in WHM and in the bandwidth section of cpanel though.

 

The "Traffic not viewed" is from spiders, such as google, rather than humans.

[Johnney]

Is There anyway to Block the Bots????? I really Don't Want My sites on the search Engines!!!

 

My Sites are Message Boards, Private By Invite Only!!!

[TCH-Don]

If your board requires registation to view

the bots cannot get in.

[TCH-Andy]

You can always add a file in your public_html directory called robots.txt. This should be a plain text file with

>User-agent: *
Disallow: /

in it. This will then stop all well behaved bots.