vendlus Posted February 21, 2004 Share Posted February 21, 2004 Inktomi Slurp This bot is by far the largest of the spiders that search my site (996 hits vs 45 by googlebot). While this is fine at the moment, I plan to open up some image galleries shortly and if this bot hits them like it is hitting my site now, it's going to rip my bandwidth apart. Here's my quesetion (well, three). Is Inktomi too important a bot for me to just ban it with a robots.txt file? If not, is this the correct format for that file? >User-agent: Inktomi Slurp Disallow: / If it is too important.... since my image galleries will all be seen only from the same image viewing template, would it work for me to just put the robot metatag into the image browser page to keep this bot out of my images? Or would I need to add each of my image folders to my robots.txt file? Or am I confused somewhere (which would not surprise me at all )? Quote Link to comment Share on other sites More sharing options...
MikeJ Posted February 21, 2004 Share Posted February 21, 2004 You could do something like I do in my robots.txt: >User-agent: * Disallow: /cgi-bin/ Disallow: /images/ Disallow: /gallery/ That should, I believe, rule out all subdirectories of the above as well. So if you can keep your gallery in it's own directory tree, it should work. Quote Link to comment Share on other sites More sharing options...
vendlus Posted February 22, 2004 Author Share Posted February 22, 2004 I can do that. Thanks. Quote Link to comment Share on other sites More sharing options...
SEO Posted February 22, 2004 Share Posted February 22, 2004 Is Inktomi too important a bot for me to just ban it with a robots.txt file? It depends on the 'point' of the site, if you want targeted viewers interested in your content... the answer would be yes. P.S. If you site is just recently added to the index they will hit you hard in the beginning but this will not be an ongoing, continuous ordeal (i.e. they will not always be "ripping your bandwidth apart"). Quote Link to comment Share on other sites More sharing options...
vendlus Posted February 22, 2004 Author Share Posted February 22, 2004 If you site is just recently added to the index they will hit you hard in the beginning but this will not be an ongoing, continuous ordeal Ok. We just opened these pages to the public, so that's good to know. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.