Jump to content

Web Crawler


Recommended Posts

hello all!


I run MySoccerLinks.com


It's a soccer specific directory.


I'm adding websites manually, as well as encouraging soccer websites to submit themselves.


I'm looking for a crawler to add websites to a 'pending' sql database, so then I can review, modify and categorize, and approve for inclusion.


I want to be able to set the starting URL, and then allow the crawler to do it's thing.


I'm not looking for millions of URLs, not looking to be the next DMOZ - I'm just looking for an easier way to add websites without a lot of the finding, cutting, and pasting that I'm doing now.



Anyone have any suggestions?




Link to comment
Share on other sites

The only thing I can think of at the moment is the PHPLinks that is located in your cPanel.


PHPlinks is not in TCH's cpanel. It comes with Fantastico but I guess TCH found it as a treat and removed it.


I could be wrong! I was the first time.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Create New...