Forums

Robots.txt and my Core Concrete5 is getting indexed by Google!!!

Permalink
Was checking out a site I updated using the control panel (creates a new folder (updates) and a line in the config/site.php pointing to the new file)
The problem is - it doesn't look like the robots.txt has the /updates folder as disallow.

Just wanted to bring that to people's attention. I assume I'm not the only one that updated this way and will soon be having my core files indexed on Google! similar to the thread below:

http://www.concrete5.org/index.php?cID=3621...

Besides adding the /updates folder to the robots.txt - does anyone have any suggestions, or should this suffice? I also think I need to turn off directory listing.

Thanks,
Phil
thephilm
 
Remo replied on at Permalink Reply
Remo
No, what else do you need? You could use a htaccess file to protect certain directories that people don't have to have access to.

But even if google indexes the updates directory, it's not going to cause a lot of problems, is it? Don't get me wrong, I'm not saying it's a good thing that robots.txt doesn't contain this directory, just saying that it's not the end of the world..

Add it to robots.txt and you're fine..

I'd actually say it's a lot more important that directory browsing is disabled!
arcanepain replied on at Permalink Reply
arcanepain
Just this instant spotted the same thing and was popping on here to flag it. Remo's right...don't think it's the biggest deal in the world, but a quick update the robots.txt should do the trick. Just done it for the 2/3 sites of mine i've currently updated to 5.4.0.5 in this way.

Probably wouldn't be a bad idea to stick this into the default robots.txt going foward...