Fatbot 2.0 - Bad Crawler

[wmadill]

One of my clients just blew through his bandwidth allotment because a crawler identified as "FatBot 2.0" from a company called "FatLens" hit his site hard for four days. Several days there were over 9,000 hits (it's a big e-commerce site) which is pretty excessive.

 

The IP addresses are 64.124.140.150 through 64.124.140.154. For much of the last three days, it was using all 5 IP addresses. I've banned the IP addresses, sent a peeved email to their contact address on their website, and scraped together enough bandwidth to put the client back online.

 

FatLens, according to their PR, has been a ticket comparison site ("find the cheapest ticket to x event") and they are rolling out comparisons for more products than just tickets. So, as a result, they are crawling more sites. Until May 25, they only occasionally visited sites I manage but something changed for this one client that day.

 

If I get any response from the company, I'll update this post.

 

Bill Madill

bill at synotac dot com

Edited May 30, 2006 by wmadill

[wmadill]

Figured it was time for an update to this post.

 

FatLens (the folks who seem to run the FatBot crawler) never replied to my email complaint. Then to add insult to injury, my client's site got hit again over the weekend with IP addresses in the same block (64.124.140.*) but different from the ones I blocked. So I blocked that entire range. And even though it is early June, the client is in a world of hurt because he's used up 93% of his assigned bandwidth for the month! I've scrapped together more to tide him over for a bit but it will be tight.

 

I also made some osCommerce changes to try minimizing the impact of this robot and next month I'll turn off the IP blocks and see what happens.

 

The site was hit about 6 times a minute for 2 straight days which does add up....

 

Bill

[TCH-Don]

Bill thanks for the update.

 

Your info will help other too.