Inktomi Slurp

[vendlus]

Inktomi Slurp

 

This bot is by far the largest of the spiders that search my site (996 hits vs 45 by googlebot). While this is fine at the moment, I plan to open up some image galleries shortly and if this bot hits them like it is hitting my site now, it's going to rip my bandwidth apart.

 

Here's my quesetion (well, three). Is Inktomi too important a bot for me to just ban it with a robots.txt file?

 

If not, is this the correct format for that file?

>User-agent: Inktomi Slurp
Disallow: /

 

If it is too important....

since my image galleries will all be seen only from the same image viewing template, would it work for me to just put the robot metatag into the image browser page to keep this bot out of my images? Or would I need to add each of my image folders to my robots.txt file? Or am I confused somewhere (which would not surprise me at all )?

[MikeJ]

You could do something like I do in my robots.txt:

 

>User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /gallery/

 

That should, I believe, rule out all subdirectories of the above as well. So if you can keep your gallery in it's own directory tree, it should work.

[vendlus]

I can do that.

 

Thanks.

[SEO]

Is Inktomi too important a bot for me to just ban it with a robots.txt file?

It depends on the 'point' of the site, if you want targeted viewers interested in your content... the answer would be yes.

 

P.S. If you site is just recently added to the index they will hit you hard in the beginning but this will not be an ongoing, continuous ordeal (i.e. they will not always be "ripping your bandwidth apart").

[vendlus]

If you site is just recently added to the index they will hit you hard in the beginning but this will not be an ongoing, continuous ordeal

Ok. We just opened these pages to the public, so that's good to know.