Jump to content

Spiders Do Cost Bandwidth


hindixp
 Share

Recommended Posts

Use a robots.txt file to disallow spiders from going through your heavy stuff that you don't really need indexed, and won't affect your ranking (like images).

 

For example:

 

>User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /gallery/

 

(if you have cgi-bin generated content pages you'd probably want to remove that line).

 

It might cut down on the bandwidth usage.

Link to comment
Share on other sites

Use a robots.txt file to disallow spiders from going through your heavy stuff that you don't really need indexed, and won't affect your ranking (like images).

 

For example:

 

>User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /gallery/

 

(if you have cgi-bin generated content pages you'd probably want to remove that line).

 

It might cut down on the bandwidth usage.

Well, it seems I've already done this. I need to come up with more.. :)

 

It's funny, how there are only a few images that come up on Google's Image Search for OMGN... not a single one is located on OMGN.

Link to comment
Share on other sites

If there are certain spiders that are "abusive" you can disallow individual spiders.

 

Particularly if they are spiders you don't care about.

 

Unfortunately, there's no magic button, though, for spiders.

Link to comment
Share on other sites

Another problem is that it's up to the spider author to be honest and respect your robots.txt file. There is no reason, other than honor, that a spider has to obey your wishes.

Link to comment
Share on other sites

Well, it seems I've already done this. I need to come up with more.. :)

 

It's funny, how there are only a few images that come up on Google's Image Search for OMGN... not a single one is located on OMGN.

Make sure you look at your logs to see which assets are being viewed and which of them are the biggies.

 

I worked for a company called the Multi Player Game Network and we were commonly called MPGN. I did a search on the abbr and it took me to a ton of sites about a strange neurological disorder by the same name. Kind of a cosmic irony there, eh? :unsure:

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...