cmuskett Posted July 16, 2003 Share Posted July 16, 2003 (edited) I just found psbot at my site psbot. I have to add something else to my robots.txt I went to the URL and this is what the page says. What is Psbot?Picsearch is indexing pictures from the web. To do this we use a web-crawler which identifies itself as 'Psbot'. Load spreading technique Picsearch utilizes a load spreading technique which reduces the load on the indexed domains' web-servers. The effect is that then the web-servers will have several minutes between requests from Picsearch. Although Picsearch is up and running, there is still development work in progress, and it is possible that a webserver will occationally have shorter intervals between requests. How to avoid being indexed If you do not wish Picsearch to index your pictures please add 'psbot' to your robots.txt file according to the The Robot Exclusion Standard (example provided below). If you do not have access to the robots.txt file Psbot also obeys the robots meta tag (example provided below). robots.txt example: Include the following in your robots.txt file: User-agent: psbot Disallow: / META tag example: Include the following in the head section of your web-page: <meta name="robots" content="noindex,nofollow"> Experiencing problems? Since Psbot is still very new we apologize for any inconvenience it may cause and would ask you to notify Picsearch about any problems at psbot@picsearch.com in order for Picsearch to rectify them as quickly as possible. Thank you. Edited July 16, 2003 by cmuskett Quote Link to comment Share on other sites More sharing options...
TCH-Don Posted July 16, 2003 Share Posted July 16, 2003 Thank you Charlotte, I have added the robots.txt file. Thumbs Up Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.