vendlus Posted February 21, 2004 Share Posted February 21, 2004 Inktomi Slurp This bot is by far the largest of the spiders that search my site (996 hits vs 45 by googlebot). While this is fine at the moment, I plan to open up some image galleries shortly and if this bot hits them like it is hitting my site now, it's going to rip my bandwidth apart. Here's my quesetion (well, three). Is Inktomi too important a bot for me to just ban it with a robots.txt file? If not, is this the correct format for that file? >User-agent: Inktomi Slurp Disallow: / If it is too important.... since my image galleries will all be seen only from the same image viewing template, would it work for me to just put the robot metatag into the image browser page to keep this bot out of my images? Or would I need to add each of my image folders to my robots.txt file? Or am I confused somewhere (which would not surprise me at all )? Quote Link to comment Share on other sites More sharing options...
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.