Family Forum Needs To Avoid Search Engines?

[McCrockett]

Hello:

I'm not really sure where to post this, so I hope this doesn't cause any problems.

 

Anyhow, I would like to put a forum on my webspace through TCH and only allow family and friends to join. My concern, however, is that search engine robots would find it because I've already submitted my main domain to several different search engines.

 

You see, for something like this - I would prefer a certain level of privacy and not have problems with random people always trying to join.

 

My brother recommended I try a subdomain that would redirect to the forum, but I'm not sure I even know how a subdomain works or even if it would avoid the search engine robots. So, my questions are the following:

 

1)Would a subdomain avoid search engine robots even after I've submitted my main domain?

 

2)Any ideas on what would be the best way of starting my own family forum?

 

Well, I hope this all makes sense. Thanks in advance for any help.

[TCH-Bruce]

If you place a link on a page in your site to direct you to the forum it will also be spidered unless you place a robots.txt file in the same folder as your forum that denies spiders from indexing it.

 

As for making the forum private, just make sure you set all the permissions on your forum divisions so that only registered members can access them.

[telcor]

If you place a link on a page in your site to direct you to the forum it will also be spidered unless you place a robots.txt file in the same folder as your forum that denies spiders from indexing it.

 

As for making the forum private, just make sure you set all the permissions on your forum divisions so that only registered members can access them.

 

Remember:

 

Well-behaved spiders/bots will only look for robots.txt in your root domain (http://www.example.com/robots.txt, rather than http://www.example.com/forum/robots.txt).

 

Poorly coded, or malicious, bots will ignore robots.txt. One "solution" to that is requiring all visitors to login, even to read threads, not just for posting.

[McCrockett]

Thanks for the replies guys. Okay, I read those documents on robots and I think I may have found the answer. But if I could, I would like to run it by you just to be sure. First I would make a file called robots.txt and put it in my root directory, so it would look something like this:

 

http://www.davidcrockettscoins.com/robots.txt

 

Then, the only stuff saved in the robots.txt would be the following:

>User-agent: *
Disallow: /forum/

 

Is this correct? Also, would that be enough to avoid indexing robots for that portion of my webspace? If so, that's a lot easier than I thought.

[telcor]

Thanks for the replies guys. Okay, I read those documents on robots and I think I may have found the answer. But if I could, I would like to run it by you just to be sure. First I would make a file called robots.txt and put it in my root directory, so it would look something like this:

 

http://www.davidcrockettscoins.com/robots.txt

 

Then, the only stuff saved in the robots.txt would be the following:

>User-agent: *
Disallow: /forum/

 

Is this correct? Also, would that be enough to avoid indexing robots for that portion of my webspace? If so, that's a lot easier than I thought.

 

 

that's it

[TCH-JimE]

Hello,

 

Correct, however, if I have heard that some "bad" robots, take the robots.txt and process the information in it and then completley disregard it.

 

In conjunction with the "robots.txt" I would also suggest that viewing of forum is not made public (you have to be registered) and to register the admin has to approve the application. All good boards have this function in it. That way, it makes life a lot easier.

 

JimE

[telcor]

Hello,

 

Correct, however, if I have heard that some "bad" robots, take the robots.txt and process the information in it and then completley disregard it.

 

In conjunction with the "robots.txt" I would also suggest that viewing of forum is not made public (you have to be registered) and to register the admin has to approve the application. All good boards have this function in it. That way, it makes life a lot easier.

 

JimE

 

yeah. robots.txt relies upon politeness, which is seriously lacking thanks to spammers and other like ilk.

 

There are other, trickier, methods of blocking bots, but requiring registration and moderator approval is the only solution that truly works.