Issue With Robot.txt

[s4suser]

Hello all,

 

I have created robot.txt for my site.earlier it was working properly.Robots was followed the rules written on robot.txt.

 

But since last few days suddenly i faced issue regarding web robots which visits pages that i have disallowed earlier in robots.txt and send me spam mail.

 

Please help me, why this happens?

[TCH-Thomas]

Welcome to the forum, s4user. :9

 

Robots.txt have nothing to do with spam mail as far as I know. This file only controls which search robots are allwed to visit your site.

 

To control email blacklisting/whitelisting you will have to use/modify spamassassin.

[s4suser]

Thanks TCH-Thomas for ur reply.

 

No actualy i have one form for sending mail whose action is .php file

I have restricted direct access to this php file using

 

Disallow: /sendmail.php

 

Still some robots are accessing these file and that result in i got empty mail.

This robots are ignoring my robots.txt and access the page.

 

This was working earlier. But nowadays i m facing this issue.

[TCH-Bruce]

First thing I would do is rename your sendmail.php file to something else, preferably without the word "mail" in it. The second thing I would do is make sure your script is secure. That should stop the mails from trying to be sent.

[OJB]

Are you sure they are search engine spiders/robots and not just spam bots?

 

Spam bots won't take notice of your robots.txt file and if your form allows anyone to submit an entry without some sort of anti-spam mechanism (reCaptcha for instance) then spambots will happily submit entries to your form.

 

I'd suggest the following:

 

1) front-end (Javascript) validation to ensure valid emails, valid names and non-empty fields are prevented

2) back-end (PHP) validation to ensure valid emails, valid names and non-empty fields are prevented

3) reCaptcha or some other variety of anti-spam mechanism (it can be as simple as a challenge and response asking a question that only a human can answer)