musicfrisk Posted May 8, 2003 Share Posted May 8, 2003 Good bot? Bad bot? Where does this bot's data ultimately get used? which search engines? Quote Link to comment Share on other sites More sharing options...
TCH-Andy Posted May 8, 2003 Share Posted May 8, 2003 musicfrisk, Good Bot I would say. Yahoo is the main search engine, but I think it also covers MSN, HotBot, Overture, LookSmart, About, Lycos, and others I think from memory. Andy Quote Link to comment Share on other sites More sharing options...
SEO Posted May 8, 2003 Share Posted May 8, 2003 Slurp is Inktomi Corporation's web-indexing robot. It collects documents from the web to build a searchable index for search services using the Inktomi search engine, including Hotbot, LookSmart, Overture and Excite. This year Yahoo bought Inktomi... what Yahoo is going to do with this purchase is unclear at this point (their primary source for search results is Google). Some in the community believe that Google will continue to supply the majority of rankings and that they will use Inktomi for the business listings (Pay Per Clicks, PPC). So, good bot. Quote Link to comment Share on other sites More sharing options...
TCH-Andy Posted May 8, 2003 Share Posted May 8, 2003 Yes, you're right about Yahoo. My mistake, I stand corrected Andy Quote Link to comment Share on other sites More sharing options...
musicfrisk Posted May 8, 2003 Author Share Posted May 8, 2003 here's my problem though, I have bots coming by from places I'm linked. But not inktomi, nor google has ever gone past that one page that brought them here. Yahoo also never has accpted my submission yet, that was key in the past sites I've been with. Everytime I get a new link I ask google to crawl THAT site...anyone else got ideas on Yahoo or google full site crawls - I've doen everything in the other threads suggested. Thanks people!!! Quote Link to comment Share on other sites More sharing options...
SEO Posted May 8, 2003 Share Posted May 8, 2003 Google does not actually use submissions anymore. There are people that will argue this point but it is quite clear to me that they do not. Their source of 'new' pages comes from their fresh crawl. Meaning an indexed site will 'feed' new sites via links. That means the best way to get crawled is to have links to your site on other external pages. Having said that, if the fresh bot has 'seen' your site... it will be back. It can take at least two cycles of a true crawl (not fresh) and dance (or two) before you are actually indexed. This use to take place every month (about the third week). The last two dances have not been as frequent (almost 6 weeks apart). So, not personally going back and reading your previous posts, has it been more then 3 months since the initial 'fresh' visit? With regards to Yahoo, you are talking about the directory submission for $299.00 a year? Quote Link to comment Share on other sites More sharing options...
musicfrisk Posted May 9, 2003 Author Share Posted May 9, 2003 Google does not actually use submissions anymore. There are people that will argue this point but it is quite clear to me that they do not. Their source of 'new' pages comes from their fresh crawl. Meaning an indexed site will 'feed' new sites via links. That means the best way to get crawled is to have links to your site on other external pages. Having said that, if the fresh bot has 'seen' your site... it will be back. It can take at least two cycles of a true crawl (not fresh) and dance (or two) before you are actually indexed. This use to take place every month (about the third week). The last two dances have not been as frequent (almost 6 weeks apart). So, not personally going back and reading your previous posts, has it been more then 3 months since the initial 'fresh' visit? With regards to Yahoo, you are talking about the directory submission for $299.00 a year? no more like one month since google hit my front page with no links before launch. The reason that concerns me is that part of the page is dynamic, I have concerns it's not reading other pages. Yahoo was a normal submission, they've usually gotten back to my within a month before with sites taht were much less credible and organized. Just frustrated!!! Quote Link to comment Share on other sites More sharing options...
SEO Posted May 9, 2003 Share Posted May 9, 2003 Yahoo was a normal submissionAre you talking about the directory submission for $299.00 a year? Just frustrated!!! I am afraid that this is like getting frustrated because water will not boil in 2 min. Quote Link to comment Share on other sites More sharing options...
greatfolios sysop Posted May 13, 2003 Share Posted May 13, 2003 dsdemmin, Your posts are great, Thank you! Bill McCord www.greatfolios.com Quote Link to comment Share on other sites More sharing options...
surefire Posted May 21, 2003 Share Posted May 21, 2003 For what it's worth... I have read a lot about various ways to get dynamic links using ? & = and so forth into static looking links. Uses mod rewrite. But you might want to wait and see how successful my ideas are before you start implementing them. I'll let you know when I get the first google deepcrawl. Quote Link to comment Share on other sites More sharing options...
SEO Posted May 22, 2003 Share Posted May 22, 2003 Jack (and others): I have looked into dynamic pages extensively with regard to search engine indexing. Without a doubt, the way to go was using mod_rewrite (for Apache). However, from Google's mouth: Writing dynamic urls as if they were static used to be the "right way" to present dynamic urls, but that's changing, at least for Google. Google is getting better about crawling dynamic urls, and we'd prefer to see dynamic urls in all their glory instead of written as if they were static. You'll see an increasing number of hosts where we could crawl deeply into a site through all kinds of dynamic urls. Google was pretty much the first to crawl dynamic urls, and we want to do it right without causing webmasters to rework their site. Of course, if you have a url with 15 parameters and only two of them actually mean anything, it's always a good idea to shorten a url whenever possible by trimming out the unneeded parameters. Changing dynamic urls to appear static will be less important over time as Google crawls dynamic urls better. We also estimate host load by taking account whether a url is dynamic or not. Keeping dynamic urls written as dynamic will help us to estimate the load for your server and keep a bot from hitting your server too often. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.