My Googlebot Phpbb Experience

[Bob Crabb]

During the last few days of June, I noticed that my bandwidth consumption had taken a sharp rise on a site where I have a very active phpBB2 forum. Since the forum had been more active than normal, I really didn't think much about it at first. Normally the site sees bandwidth consumption of 200 - 300 MB per day, but now it was using well over 1GB/day. Looking into AWstats, I saw that Googlebot was eating about 1GB/day.

 

Good grief!!! I started doing research on this forum, and on others about how to slow it down. I have had the forum for almost two years, and most of the script was already disallowed in my robots.txt. Checking the latest visitor log, I noticed something interesting. Googlebot was looking at a script called calendar.php, and was methodically looking at every day in the calendar over a 10 year period. About a month before, I had added a calendar mod to the forum, and had NOT added calendar.php to the disallow list in robots.txt.

 

Problem solved, I thought. Adding it to robots.txt had no effect. I even went so far as to disallow all files, which had no effect. I was about to block googlebot from the site altogether via cPanel, but instead decided to ban the googlebot IP address from the forum. That worked!!! After all bot activity subsided, I restored access to the rest of the site via robots.txt, and after a week, removed the googlebot IP addy from the forum banned IP list.

 

Everything seems to be back to normal now.

 

This site is set up in an account where I normally have plenty of extra bandwidth, so I wasn't too concerned about going over the limit, but I recently set up a phpBB2 forum with a calendar mod on another site that has a much more modest bandwidth allotment, and I was very concerned about something similar happening on that site. Anyway, I now at least think that I know how to deal with it.

[TCH-Don]

Thanks for the info Bob.