Jump to content

Inktomi Slurp


vendlus

Recommended Posts

Inktomi Slurp

 

This bot is by far the largest of the spiders that search my site (996 hits vs 45 by googlebot). While this is fine at the moment, I plan to open up some image galleries shortly and if this bot hits them like it is hitting my site now, it's going to rip my bandwidth apart.

 

Here's my quesetion (well, three). Is Inktomi too important a bot for me to just ban it with a robots.txt file?

 

If not, is this the correct format for that file?

>User-agent: Inktomi Slurp
Disallow: /

 

If it is too important....

since my image galleries will all be seen only from the same image viewing template, would it work for me to just put the robot metatag into the image browser page to keep this bot out of my images? Or would I need to add each of my image folders to my robots.txt file? Or am I confused somewhere (which would not surprise me at all B))?

Link to comment
Share on other sites

You could do something like I do in my robots.txt:

 

>User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /gallery/

 

That should, I believe, rule out all subdirectories of the above as well. So if you can keep your gallery in it's own directory tree, it should work.

Link to comment
Share on other sites

Is Inktomi too important a bot for me to just ban it with a robots.txt file?
It depends on the 'point' of the site, if you want targeted viewers interested in your content... the answer would be yes.

 

P.S. If you site is just recently added to the index they will hit you hard in the beginning but this will not be an ongoing, continuous ordeal (i.e. they will not always be "ripping your bandwidth apart").

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Unfortunately, your content contains terms that we do not allow. Please edit your content to remove the highlighted words below.
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...