Kevan Posted August 19, 2008 Share Posted August 19, 2008 I notice that a new web crawler, its a pic dot com, is pulling message posts repeatedly from my SMF forums. Every hour pulling data at a rate 2 times my normal daily traffic. (I submitted a ticket to support already). It looks like it changes the message every minute. (Pulling the same message about 20 times per minute. I don't know if anyone else is experiencing the same issue or not but a quick look at my bandwidth report was a shocker. Does anyone have any info about this thing? Quote Link to comment Share on other sites More sharing options...
MikeJ Posted August 19, 2008 Share Posted August 19, 2008 You could try contacting the owner of the crawler if it's misbehaving: http://itsapic.com/crawler.html itsapic.com crawler is designed to respect the robots.txt exclusion directives and META robots tags, and collect material at a measured, adaptive pace unlikely to disrupt normal website activity. Our crawler identification includes itsapic.com_crawler. If you notice our crawler behaving poorly please send us email at the address as indicated in crawler identification string. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.