Google crawl errors
Permalink
I'm seeing a lot of crawl errors from Google, when it tries to get into places like this:
http://creditreporthowto.us/updates/concrete5.4.2.1/concrete/themes...
I'm not sure where it's finding an entry point intohttp://creditreporthowto.us/updates,... buy I think having the robots.txt file exclude the /updates directory will address the problem.
It would be great if the robots.txt file generated by the base product could include this.
http://creditreporthowto.us/updates/concrete5.4.2.1/concrete/themes...
I'm not sure where it's finding an entry point intohttp://creditreporthowto.us/updates,... buy I think having the robots.txt file exclude the /updates directory will address the problem.
It would be great if the robots.txt file generated by the base product could include this.
Disallow: /updates