MelH Posted May 30, 2006 Share Posted May 30, 2006 hello all! I run MySoccerLinks.com It's a soccer specific directory. I'm adding websites manually, as well as encouraging soccer websites to submit themselves. I'm looking for a crawler to add websites to a 'pending' sql database, so then I can review, modify and categorize, and approve for inclusion. I want to be able to set the starting URL, and then allow the crawler to do it's thing. I'm not looking for millions of URLs, not looking to be the next DMOZ - I'm just looking for an easier way to add websites without a lot of the finding, cutting, and pasting that I'm doing now. Anyone have any suggestions? Mel Quote Link to comment Share on other sites More sharing options...
j2k4b Posted May 31, 2006 Share Posted May 31, 2006 The only thing I can think of at the moment is the PHPLinks that is located in your cPanel. Quote Link to comment Share on other sites More sharing options...
j2k4b Posted May 31, 2006 Share Posted May 31, 2006 The only thing I can think of at the moment is the PHPLinks that is located in your cPanel. PHPlinks is not in TCH's cpanel. It comes with Fantastico but I guess TCH found it as a treat and removed it. I could be wrong! I was the first time. Quote Link to comment Share on other sites More sharing options...
Deverill Posted May 31, 2006 Share Posted May 31, 2006 I've never used any of this kind of thing but I did find several interesting looking possibilities on a Google search. Quote Link to comment Share on other sites More sharing options...
MelH Posted May 31, 2006 Author Share Posted May 31, 2006 I found the crawler portion of http://www.cs.ioc.ee/~ando/sphider/ to do exactly what I wanted. Very very interesting...... Mel Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.