donbevis Posted February 19, 2008 Share Posted February 19, 2008 I was suspended twice today, and currently still am (awaiting someone processing the Ticket). After a long search for why MY code was broken, it's turned out to be because I was being hammered every second of every hour of every day since the first of the month. It wasn't my code at all. It wasn't my fault, but they caused my account to get suspended. I highly recommend everyone look at this page: http://cuill.com/twiceler/robot.html and add the IP addresses shown there into their cpanel 'IP deny' feature, as well as a robots.txt that includes: User-agent: Twiceler Disallow: / Crawl-delay: 120 They were not initially even obeying robots.txt, but that supposedly has been resolved. cuill.com is a bad player in the web crawler world, it seems. I invite you to google their name and see. But regardless, protect yourself before you suffer the fate I did today. Quote Link to comment Share on other sites More sharing options...
TCH-Dick Posted February 23, 2008 Share Posted February 23, 2008 This bot has been nothing but bad news for the past 2 years and from my recent tests it still does not obey robots.txt. I have discovered this bot nailing 3 different servers over the last 3 days for hours at a time on individual sites. The funny thing is the site on each server was very small(less than 20 pages each), yet this bot crawled it for hours. They have been claiming to be an experimental bot for a great new search engine for some time now, but have mostly caused site owners grief. I also read that at least on of the developers is an ex Google employee. In my opinion, if Twiceler is representative of his work at Google then it is no wonder he is an EX employee. As of now all know Twiceler IPs have been banned across our server farm. This ban will remain in place until I see proof that this bot is legitimate and does something other than bring down our customer sites. Quote Link to comment Share on other sites More sharing options...
TCH-Thomas Posted February 24, 2008 Share Posted February 24, 2008 You should be able to remove it. Quote Link to comment Share on other sites More sharing options...
dirtvoyles Posted February 27, 2008 Share Posted February 27, 2008 Thanks for the info, and TCH - thanks for taking care of it for us. Quote Link to comment Share on other sites More sharing options...
donbevis Posted February 29, 2008 Author Share Posted February 29, 2008 Thanks for the info, and TCH - thanks for taking care of it for us. 38.99.13.123 is now in my logs as having Twiceler for the user agent. Quote Link to comment Share on other sites More sharing options...
accident Posted March 17, 2008 Share Posted March 17, 2008 I dont know if you guys unblocked it, but it has been crawling my website for a couple hours. I did a search on the IP, found out it was this bot, and searched in google... Funny seeing my hosting company in the first 2 pages of search results. I was getting hammered by IP 38.99.44.104 I don't care too much either way since I don't use all my bandwidth anyways. But I may block it Quote Link to comment Share on other sites More sharing options...
dirtvoyles Posted March 17, 2008 Share Posted March 17, 2008 It's getting me too, but I haven't checked the IP yet. Quote Link to comment Share on other sites More sharing options...
TCH-Dick Posted March 17, 2008 Share Posted March 17, 2008 Fixed Quote Link to comment Share on other sites More sharing options...
dirtvoyles Posted March 21, 2008 Share Posted March 21, 2008 I haven't seen any returns since the fix-y. Quote Link to comment Share on other sites More sharing options...
dirtvoyles Posted April 9, 2008 Share Posted April 9, 2008 Sorry to revisit this, but I am getting hit again. I just noticed this as I was looking at blog stats... April 9, 2008 07:35:11 Twiceler Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) Sadly, I cannot find the IP, and cPanel's port is blocked at work. Quote Link to comment Share on other sites More sharing options...
laurin1 Posted April 27, 2008 Share Posted April 27, 2008 This bot has been nothing but bad news for the past 2 years and from my recent tests it still does not obey robots.txt. I have discovered this bot nailing 3 different servers over the last 3 days for hours at a time on individual sites. The funny thing is the site on each server was very small(less than 20 pages each), yet this bot crawled it for hours. They have been claiming to be an experimental bot for a great new search engine for some time now, but have mostly caused site owners grief. I also read that at least on of the developers is an ex Google employee. In my opinion, if Twiceler is representative of his work at Google then it is no wonder he is an EX employee. As of now all know Twiceler IPs have been banned across our server farm. This ban will remain in place until I see proof that this bot is legitimate and does something other than bring down our customer sites. I'm getting hammered by this bot. Has this ban been lifted? Quote Link to comment Share on other sites More sharing options...
TCH-Thomas Posted April 27, 2008 Share Posted April 27, 2008 I don´t think the ban has been lifted, more likely is that all ip´s this bot uses are not known. Quote Link to comment Share on other sites More sharing options...
laurin1 Posted April 27, 2008 Share Posted April 27, 2008 38.99.44.102 - - [25/Apr/2008:01:17:36 -0400] "GET /index.php?module=Event%20Calendar&func=view&tplview=&viewtype=day&Date=19951107&pc_username=&pc_category=&pc_topic= HTTP/1.0" 200 55099 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)" 38.99.13.121 - - [25/Apr/2008:01:17:40 -0400] "GET /index.php?module=Event%20Calendar&func=view&tplview=&viewtype=day&Date=20171204&pc_username=&pc_category=&pc_topic=&print= HTTP/1.0" 200 60395 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)" 38.99.13.123 - - [25/Apr/2008:01:17:42 -0400] "GET /index.php?module=Event%20Calendar&func=view&tplview=&viewtype=day&Date=20110831&pc_username=&pc_category=&pc_topic=&print= HTTP/1.0" 200 55318 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)" 38.99.44.103 - - [25/Apr/2008:01:17:53 -0400] "GET /index.php?module=Event%20Calendar&func=view&tplview=&viewtype=day&Date=19700603&pc_username=&pc_category=&pc_topic=&print=1 HTTP/1.0" 200 3307 "-" "Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)" Quote Link to comment Share on other sites More sharing options...
TCH-Thomas Posted April 27, 2008 Share Posted April 27, 2008 Thanks Laurin. I will ask Dick to take a look at this. Quote Link to comment Share on other sites More sharing options...
TCH-Dick Posted April 27, 2008 Share Posted April 27, 2008 This has been corrected. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.