Greedy Bot - Omni Explorer

[clayartisan]

Just wanted to ask if anyone else here has seen this and report in on my experience with it.

 

On Thur, Aug 18, my bandwidth took a huge jump - 7 times normal. In Awstats, I saw two big draws from two IP addresses. Fortunately, I had been logging the raw access logs.

 

In the log, I found this bot running all over my website and it's called OmniExplorer_bot. I found their website and it's a bare basic page that says this about who they are:

 

"Omni-Explorer is a venture-backed startup based in Silicon Valley. Stay tuned to this site; we plan on launching shortly."

 

Now, below that, they admitted that they knew they were having problems with their greedy little bots and listed their IP range for blocking. The little monster is now disallowed in my robots.txt and all of their IP's are blocked.

 

Further internet research about OmniExplorer yielded this:

 

"...Naval Ravikant as one of the execs. Ravikant, among other things, founded Epinions, which became Shopping.com, which was just bought by eBay."

 

In Awstats, it isn't identified in the Robots/Spiders section.

 

Anybody else know anything about this or have experienced it? If not, I've now got it in the messages in case any one else is wondering about it.

 

Cheers and Death to Greedy Little Bots Who Don't Behave Nicely,

Patti

[TweezerMan]

Keep in mind this is just my personal opinion (based on experience): The OmniExplorer bot is a bad bot, and you should not implictly trust or believe what is on the OmniExplorer web site about that bot.

 

1) The OmniExplorer bot has been reported to not obey robots.txt files.

2) The OmniExplorer bot has been seen using a much larger range of IP addresses than what is reported on the OmniExplorer web site.

3) The OmniExplorer bot is known to rapdily consume excessive bandwidth.

4) Any bot that misbehaves so badly should have been immediately terminated, rather than allowed to run wild on the internet.

 

I have blocked the OmniExplorer bot in my robots.txt file, but since I don't trust the OmniExplorer bot to obey it, I've also blocked it both by user agent and by IP addresses in my .htaccess file:

># Bad bot - OmniExplorer
SetEnvIfNoCase User-Agent "^OmniExplorer" bad_bot
deny from 64.62.175.133
deny from 64.62.175.134
deny from 64.71.131.110
deny from 64.71.131.114
deny from 64.71.131.117
deny from 64.71.131.121
deny from 65.19.134.7

# Block 65.19.150.192 - 65.19.150.255 (64 IPs) - OmniExplorer bot
deny from 65.19.150.192/26

# Block 65.19.169.192 - 65.19.169.255 (64 IPs) - OmniExplorer bot
deny from 65.19.169.192/26

deny from env=bad_bot

I don't know if the IP address list is complete - it's what I compiled a while back from my own server logs and possibly some other internet sources.

 

TCH forum member annie also has an entry on her Spamhuntress wiki about the OmniExplorer bot.

 

Hope this helps...

[sitesure]

I've seen a similiar single day, single address surge, though awstat is showing a different IP, and I've not examined the logs:

 

66-194-55-242.gen.twtelecom.net pages 5004 hits 5004 bandwidth 68.82 MB

 

all on 8/15.

[clayartisan]

Thanks very much for the info, David! I'm not quite up to speed on messing around with .htaccess files yet. I'll have to devote a couple hours to studying that before I touch it. Probably sooner than later.

 

Here is the email response I received from someone named Anton Stanley(at their website), in response to the email I sent them:

 

- Patti,

 

Thank you for writing in, and providing us with the specific details. I filed

a ticket with our engineers, and they are working on fixing this bug. In the

meanwhile, if you let me know your domain, I can add it to our do-not-crawl

list. We appoplogize for any trouble caused.

 

- Anton

 

Think I should trust this? I haven't answer anything back yet.

 

Again, thanks for the code sample for when I'm ready to roll up my sleeve.

[clayartisan]

Sitesure:

 

The IP you listed is different than any they list. But guess that doesn't mean much.

 

Just be sure and turn on you Raw Access logs in C-Panel. I didn't even know I had to do that until about a month ago (or maybe I accidentally turned it off when I was new). I'm glad I had the logging on, because it showed right up in it.

 

G'luck!

[TweezerMan]

Think I should trust this?  I haven't answer anything back yet.

<{POST_SNAPBACK}>

Honestly, I would not. The bot has already wasted a bunch of bandwidth on your site, it is not a well-behaved bot, and they've had months to fix problems with the bot and apparently have not done so. I wouldn't give them a second chance or the benefit of doubt, but you can make your own decision.

[clayartisan]

Well, I certainly am not going to give them that help. It they would train the bad little thing right, they wouldn't need my number. It's their problem. Thanks for the input.