Jump to content

Issue With Robot.txt


Recommended Posts

Hello all,

 

I have created robot.txt for my site.earlier it was working properly.Robots was followed the rules written on robot.txt.

 

But since last few days suddenly i faced issue regarding web robots which visits pages that i have disallowed earlier in robots.txt and send me spam mail.

 

Please help me, why this happens?

Link to comment
Share on other sites

Welcome to the forum, s4user. :9

 

Robots.txt have nothing to do with spam mail as far as I know. This file only controls which search robots are allwed to visit your site.

 

To control email blacklisting/whitelisting you will have to use/modify spamassassin.

Link to comment
Share on other sites

Thanks TCH-Thomas for ur reply.

 

No actualy i have one form for sending mail whose action is .php file

I have restricted direct access to this php file using

 

Disallow: /sendmail.php

 

Still some robots are accessing these file and that result in i got empty mail.

This robots are ignoring my robots.txt and access the page.

 

This was working earlier. But nowadays i m facing this issue.

Link to comment
Share on other sites

Are you sure they are search engine spiders/robots and not just spam bots?

 

Spam bots won't take notice of your robots.txt file and if your form allows anyone to submit an entry without some sort of anti-spam mechanism (reCaptcha for instance) then spambots will happily submit entries to your form.

 

I'd suggest the following:

 

1) front-end (Javascript) validation to ensure valid emails, valid names and non-empty fields are prevented

2) back-end (PHP) validation to ensure valid emails, valid names and non-empty fields are prevented

3) reCaptcha or some other variety of anti-spam mechanism (it can be as simple as a challenge and response asking a question that only a human can answer)

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...