bellringr Posted November 2, 2004 Posted November 2, 2004 First, I want to say how much I like the changes in the Latest Visitors. Having them aranged by IP number is very handy. Question - I have a robots.txt file set up to keep bots from crawling my blogs, guestbook, and webcam. Google, Yahoo, and all the regulars seem to be honoring that. However, I got a visit today by h*tp://www.almaden.ibm.com/cs/crawler [bc4], and even though they claim to adhere by the "rules," they crawled every folder I told them not to. Is anyone familiar with this? Is it really IBM? Is it worth sending a nasty e-mail? Quote
TCH-Thomas Posted November 2, 2004 Posted November 2, 2004 (edited) Not sure if this will help but, try: http://www.almaden.ibm.com/cs/crawler/, same link as you posted, but gives some info. Edited November 2, 2004 by TCH-Thomas Quote
TCH-Rob Posted November 2, 2004 Posted November 2, 2004 Is anyone familiar with this? Is it really IBM? Is it worth sending a nasty e-mail? No, yes and I work for them so it may go unanswered. Give it a go anyways. Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.