Crawl Errors

Permalink
Any ideas about how to prevent these crawl errors? (pls see attachment)

Thanks for your input.

1 Attachment

 
madeforspace replied on at Permalink Reply
madeforspace
500 is normally a server side error.
Do you have a robot.txt file on your server? Is it in the root?
The files Google are trying to access all seem to be ones that should be excluded by the robot.txt file.
Have a look and check you have the following in your file:

User-agent: *
Disallow: /blocks
Disallow: /concrete
Disallow: /config
Disallow: /controllers
Disallow: /css
Disallow: /elements
Disallow: /helpers
Disallow: /jobs
Disallow: /js
Disallow: /languages
Disallow: /libraries
Disallow: /mail
Disallow: /models
Disallow: /packages
Disallow: /single_pages
Disallow: /themes
Disallow: /tools

I also noticed that your URL's all have Concrete 5.4.1.1 in them, although set ups can be different you may want to look at moving your files. I would normally expect not to see Concrete 5.4.1.1 in the URL. Suggest taking out all the folder/files from the Concrete5.4.1.1 folder and putting them in your root.
If your robot.txt file is in the same folder that may be why Google can't see it.

Hope that helps
Mnkras replied on at Permalink Reply
Mnkras
yo forgot /updates

User-agent: *
Disallow: /blocks 
Disallow: /concrete 
Disallow: /config 
Disallow: /controllers 
Disallow: /css 
Disallow: /elements 
Disallow: /helpers 
Disallow: /jobs 
Disallow: /js 
Disallow: /languages 
Disallow: /libraries 
Disallow: /mail 
Disallow: /models 
Disallow: /packages


that should fix it
madeforspace replied on at Permalink Reply
madeforspace
Correctomundo, well spotted Mnkras.
They are all indeed related to the /updates.
Ignore the comments about folder location.
irussell replied on at Permalink Reply
Mnkras,

Thanks for your help. So then just to clarifiy I should insert this code into the robot.txt file right?

Where can I find it? And what would be the steps to update the robot.txt file. In another post you mention the robot.txt file is built into concrete5, but I can't seem to locate it.
Mnkras replied on at Permalink Reply
Mnkras
It should be in the root of your site, where the index.php is. If its not
there just make it.
On Mar 11, 2011 12:33 PM, "Concrete5 Community" <discussions@concretecms.com>
wrote:
irussell replied on at Permalink Reply 2 Attachments
Ok, I added the files to the robot.txt file as suggested. You can see the before and after screen shots. Could you please double check my work to see if I did this correctly?

Thanks.