jme574 Posted October 23, 2004 Share Posted October 23, 2004 According to my awstats i have 132 404 errors for the month of oct. so far. When i clicked to see what the errors are i went and check the pages that are listed but most of them work - the biggest one was /robots.txt. (with a total of 63) Does anyone know why this might be happening? Quote Link to comment Share on other sites More sharing options...
TCH-Don Posted October 23, 2004 Share Posted October 23, 2004 When search engines crawl your site, they look for robots.txt to tell them what they can index and what not to index. When they do not find it, you see the 404 (missing) error. You can put a blank text filenamed robots.txt in your public_html folder to eliminate these errors. Or you can create a robot file by using a Robots.txt file generator. Quote Link to comment Share on other sites More sharing options...
jme574 Posted October 23, 2004 Author Share Posted October 23, 2004 ok - i used the gen. you sent the link for and created a robot.txt file and saved it as such in microsoft word. Now if i understand right, i need to upload it via ftp to my root directory where my index file is located, right? I don't have a page named index, but i do have one named home, would that be the same? It is located in my public_html folder so is that where i should upload the robot.txt file to? Quote Link to comment Share on other sites More sharing options...
TCH-Andy Posted October 23, 2004 Share Posted October 23, 2004 Yes, upload it to that directory. Make sure it's a text file though - not a word document (I'd use notepad rather than word) Quote Link to comment Share on other sites More sharing options...
signatures Posted October 31, 2004 Share Posted October 31, 2004 Can this be done if i am using frontpage? can i just paste the robots.txt file on my index page, or somewhere else and then save it in my website from there? hope this makes sense...thanks, dave Quote Link to comment Share on other sites More sharing options...
TCH-Rob Posted October 31, 2004 Share Posted October 31, 2004 I wouldnt, FP and other editors put all the HTML code in there as well. This is nothing more than a simple text file. If you dont have an FTP program you can use the file manager in cPanel. Quote Link to comment Share on other sites More sharing options...
signatures Posted October 31, 2004 Share Posted October 31, 2004 OK, so just use the ftp in CP and upload to my index.htm? haven't used the ftp yet, have just changed things from frontpage....guess i'll have to learn a few more things....dave Quote Link to comment Share on other sites More sharing options...
TCH-Andy Posted October 31, 2004 Share Posted October 31, 2004 Hi Dave. The robots.txt file is slightly different to all the stuff Frontpage does. I would just loginto your cpanel, and go to the file manager. Navigate to the public_html directory, Click on "Create new file" and on the right you should see the space for a file name, and a selection box. Type robots.txt in for the filename, and the file type should be text file That's it. You do not need to edit it or anything, a blank file will be fine. Quote Link to comment Share on other sites More sharing options...
signatures Posted November 3, 2004 Share Posted November 3, 2004 OK, i took care of that, maybe it will cut down on some of my 404 hits? anything else i could do? thanks for the help...dave Quote Link to comment Share on other sites More sharing options...
signatures Posted November 6, 2004 Share Posted November 6, 2004 I still receive quite a few 404 hits, is there anything else to do to cut down or is it even necessary to? already have 120 first 5 days of the month. thanks, dave Quote Link to comment Share on other sites More sharing options...
TCH-Bruce Posted November 6, 2004 Share Posted November 6, 2004 What is generating the 404 errors? Some 404 errors you cannot stop. Robots or other bots looking for scripts on your site that don't exist are going to give you a 404 error every time. You can't block them or stop them. The only 404s you need to concern yourself with are for content (pages, images, scripts...) you actually created that may be named wrong in a link on one or several of your web pages. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.