Forums

Exclude pages from search engines

Permalink
If I want to exclude certain pages from search engines and I am using a google webmaster account,
Do I need to exclude from the search index and the sitemap.xml both for those pages that I wish to exclude?

Is there anything more I need to do?
 
mkly replied on at Permalink Reply
mkly
AFAIK you need to exclude these in your robots.txt file. I don't think there is a way to do it through the dashboard. There is a file called robots.txt in the root of your site. Edit it to add additional areas to exclude from search spiders.

You should probably take into account both pretty URLs and (un)pretty URLs with something like this

Disallow: /index.php/example_directory
Disallow: /example_directory
synergeticrealty replied on at Permalink Reply
I only want to exclude certain pages.
Would I have to put each page individually in the robot file?

When you do the exclude from Search Index it adds this line into the header <meta name="robots" contents="noindex" />

Does that do the same thing as editing the robot file or not?
mkly replied on at Permalink Best Answer Reply
mkly
Looks like the forum ate your tag. But if you are talking about
<meta name="robots" content="noindex" />


Then yes, according to Google that should work

http://support.google.com/webmasters/bin/answer.py?hl=en&answer...
synergeticrealty replied on at Permalink Reply
Thanks for the link.
And yes that was the code I was talking about.

Thanks for the input.
Steevb replied on at Permalink Reply
Steevb
Exclude pages from sitemap.xml, but not search index if you want to search your site.

You can prevent engines further by adding 'disallow' to your robot.txt file.