Website

[jme574]

I have been told by several people to NOT use the automatic search engine submit tools and that if i do, it could have a negitive effect and could possibly cause my url's to be banner by some of the search engines.

 

my sites seem to only be found or crawled by one bot and that is the msn bot. So here is the question. If i am to submit my sites manually, who has a list of the best search engines to submit to?

 

Can we create a running lists of search engines so that not only I, but any other people that might need this info, could have a quick list to pull from?

 

Thank you for any help!

[jme574]

When adding a robot.txt to your directory, should it be a robots.txt folder or file and it should be placed in the public_html folder, right?

 

Also, i used a robot text gen. and this is what it has created:

 

# Robots.txt file created by http://www.webtoolcentral.com

# For domain: http://totallymobiledjs.com

 

# All robots will spider the domain

User-agent: *

Disallow:

 

is that good enough for a robot.txt file? is there anything i can add that will help draw crawlers and bots to my site?

Edited March 7, 2005 by jme574

[TCH-Bruce]

Adding a robots.txt file to your domain is not going to bring more search engine crawlers to you. The robots.txt file is for the bots to read and not index things you add to the file. Most search engines behave and honor the robots.txt file while some do not.

 

To get the search engines crawling your site you need incoming links to your site. So that when the site linking to you gets crawled it will lead the bot to your site.

 

There are many search engine out there. Some more popular than others. You mention MSN, there is also Google, Yahoo, Netscape, AOL, NorhternLights, Hotbot and a slew of others.

[Thumper]

You can check out http://www.submitplus.com/freeprograms.php?b=top10 I've used it before, and seemed to get some bots crawling.

[TCH-Dick]

dmoz Open Directory Project

Google

Yahoo

- need to sign up for free yahoo account

MSN Search

[Deverill]

This is an oversimplification, but now days the free submits don't usually do anything useful... even if you go to the engine itself. The only good and reliable way to get the search engines to notice you is to get links onto pages of sites that they are already visiting on a regular basis.

[jme574]

The only good and reliable way to get the search engines to notice you is to get links onto pages of sites that they are already visiting on a regular basis.

To get the search engines crawling your site you need incoming links to your site. So that when the site linking to you gets crawled it will lead the bot to your site.

 

 

Ok - this leads me to another question. If i have say 7 other sites that i control or have access to, would it work if i added a page under the public_html folder that had links to the other websites. Instead of having to redo a website to add a link into the existing site. That way even though a regular person surfing the site wouldn't actually see the page with links but a bot or spider crawling the public_html folder would find the page and crawl to the sites from there?

[TCH-Bruce]

If a person cannot navigate to the page then the bot can't either. All a bot does is follow links.

 

You could add a site map page to your site and link to it from you home page. In the site map link to the page with links to your other sites.

[Deverill]

Exactly - bots just act like surfers to see where they can get. They don't have the ability to scan your public_html folder.

 

Another thing to keep in mind is that some search engines do not pay much attention to pages with the word "links" in the filename so use "partners" "other-sites" or something like that.

 

Finally, although linking a set of sites together will help some, especially if you have one that the engines are visiting and you want to send them to the other site, they do notice if it's just a small number of sites all linking to each other.

 

It is vital to get links to your site from other, preferably related, sites and the more the better for the engines.

 

Important: I said before that sumbitting to search engines is all but useless, and that is true, but that does not apply to directories such as DMOZ. The directories are usually human edited and you should submit to them if you want to be listed. The search engines have bots but the directories don't and thus will not know about you without someone submitting your site.

 

Just wanted to make that clear.