hindixp Posted March 16, 2004 Posted March 16, 2004 Hi, my stats show a usage of Robots/Spiders visitors Jeeves hits 8496 Bandwidth 151.66 MB How to Save this without loosing the Ranking. See ya, manu Quote
DarqFlare Posted March 16, 2004 Posted March 16, 2004 I'm trying to come up with a solution myself... Quote
MikeJ Posted March 16, 2004 Posted March 16, 2004 Use a robots.txt file to disallow spiders from going through your heavy stuff that you don't really need indexed, and won't affect your ranking (like images). For example: >User-agent: * Disallow: /cgi-bin/ Disallow: /images/ Disallow: /gallery/ (if you have cgi-bin generated content pages you'd probably want to remove that line). It might cut down on the bandwidth usage. Quote
hindixp Posted March 16, 2004 Author Posted March 16, 2004 Thanks for the example. See ya, manu Quote
sts Posted March 16, 2004 Posted March 16, 2004 Is there any way to find out who is Unknown robot (identified by 'robot') or Unknown robot (identified by 'spider')? Quote
DarqFlare Posted March 16, 2004 Posted March 16, 2004 Use a robots.txt file to disallow spiders from going through your heavy stuff that you don't really need indexed, and won't affect your ranking (like images). For example: >User-agent: * Disallow: /cgi-bin/ Disallow: /images/ Disallow: /gallery/ (if you have cgi-bin generated content pages you'd probably want to remove that line). It might cut down on the bandwidth usage. Well, it seems I've already done this. I need to come up with more.. It's funny, how there are only a few images that come up on Google's Image Search for OMGN... not a single one is located on OMGN. Quote
MikeJ Posted March 16, 2004 Posted March 16, 2004 If there are certain spiders that are "abusive" you can disallow individual spiders. Particularly if they are spiders you don't care about. Unfortunately, there's no magic button, though, for spiders. Quote
Deverill Posted March 16, 2004 Posted March 16, 2004 Another problem is that it's up to the spider author to be honest and respect your robots.txt file. There is no reason, other than honor, that a spider has to obey your wishes. Quote
Deverill Posted March 17, 2004 Posted March 17, 2004 Well, it seems I've already done this. I need to come up with more.. It's funny, how there are only a few images that come up on Google's Image Search for OMGN... not a single one is located on OMGN. Make sure you look at your logs to see which assets are being viewed and which of them are the biggies. I worked for a company called the Multi Player Game Network and we were commonly called MPGN. I did a search on the abbr and it took me to a ton of sites about a strange neurological disorder by the same name. Kind of a cosmic irony there, eh? Quote
DarqFlare Posted March 17, 2004 Posted March 17, 2004 haha, nice! There isn't much using the acronym OMGN actually... Some mathmatical equations use it, some people use it for like "oh my god", but I have no idea what the 'n' would stand for... Quote
hindixp Posted March 17, 2004 Author Posted March 17, 2004 That "n" was like a addon domain to acronym known to me. see ya, manu Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.