Jump to content
Sign in to follow this  
hindixp

Spiders Do Cost Bandwidth

Recommended Posts

Hi,

my stats show a usage of Robots/Spiders visitors

Jeeves hits 8496 Bandwidth 151.66 MB

 

How to Save this without loosing the Ranking.

 

See ya,

manu

Share this post


Link to post
Share on other sites

Use a robots.txt file to disallow spiders from going through your heavy stuff that you don't really need indexed, and won't affect your ranking (like images).

 

For example:

 

>User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /gallery/

 

(if you have cgi-bin generated content pages you'd probably want to remove that line).

 

It might cut down on the bandwidth usage.

Share this post


Link to post
Share on other sites

Is there any way to find out who is Unknown robot (identified by 'robot') or Unknown robot (identified by 'spider')?

Share this post


Link to post
Share on other sites
Use a robots.txt file to disallow spiders from going through your heavy stuff that you don't really need indexed, and won't affect your ranking (like images).

 

For example:

 

>User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /gallery/

 

(if you have cgi-bin generated content pages you'd probably want to remove that line).

 

It might cut down on the bandwidth usage.

Well, it seems I've already done this. I need to come up with more.. :)

 

It's funny, how there are only a few images that come up on Google's Image Search for OMGN... not a single one is located on OMGN.

Share this post


Link to post
Share on other sites

If there are certain spiders that are "abusive" you can disallow individual spiders.

 

Particularly if they are spiders you don't care about.

 

Unfortunately, there's no magic button, though, for spiders.

Share this post


Link to post
Share on other sites

Another problem is that it's up to the spider author to be honest and respect your robots.txt file. There is no reason, other than honor, that a spider has to obey your wishes.

Share this post


Link to post
Share on other sites
Well, it seems I've already done this. I need to come up with more.. :)

 

It's funny, how there are only a few images that come up on Google's Image Search for OMGN... not a single one is located on OMGN.

Make sure you look at your logs to see which assets are being viewed and which of them are the biggies.

 

I worked for a company called the Multi Player Game Network and we were commonly called MPGN. I did a search on the abbr and it took me to a ton of sites about a strange neurological disorder by the same name. Kind of a cosmic irony there, eh? :unsure:

Share this post


Link to post
Share on other sites

haha, nice!

 

There isn't much using the acronym OMGN actually... Some mathmatical equations use it, some people use it for like "oh my god", but I have no idea what the 'n' would stand for...

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...