Awstats

[Miriam]

I finally got around to exploring awstats, and was really impressed with the information available. But I have a questions about the reporting on robots.

 

I found an entry for May 8:

Unknown robot (identified by 'robot') hits: 320+2 bandwidth 43.79 MB

 

This robot used a tremendous amount of bandwidth. I would like to exclude it in my robots.txt file, but since it only identified by 'robot', I don't know how to write the command. Could someone tell me how I can exclude this particular unknown robot?

 

Thanks

Miriam

[TCH-Thomas]

Not sure if it will help, but have a look at

http://www.robotstxt.org/wc/norobots.html

[TCH-Dick]

Use the information from AWStats and Latest Vistors to identify a bot. the following is from my cPanel.

 

Awstats:

Unknown robot (identified by hit on 'robots.txt') 0+6 1.44 KB 24 May 2004 - 08:52

 

Latest Visitors:

Host: 65.54.164.79

Url: /

Http Code : 200

Date: May 24 08:52:59

Http Version: HTTP/1.0"

Size in Bytes: 37995

Referer: -

Agent: msnbot/0.11 (+http://search.msn.com/msnbot.htm)

 

 

Now I have all the information I need to exclude this robot.

[TCH-Bruce]

Moved for orgainization