The robots meta tag is used to control how search engines treat the page and the links on that page. The tag allows you to choose whether to have search engines “index” or “noindex” the page and whether or not to “follow” or “nofollow” the links on the page.

The robots meta tag is very powerful in that it can be used to prevent a page from being indexed in search engines. According to Google “When we see a noindex meta tag on a page, Google will completely drop the page from our search results, even if other pages link to it”.

Normally when used the robots meta tag will look like:

<meta name="robots" content="index, follow" />

If you want to prevent a page from being included in the search engines use

<meta name="robots" content="noindex, nofollow" />

Additional Robots Tags

There are a couple of other operators that can be used with the robots meta tag if desired. If you choose to use one or more of these they are each used separately in their own tag.

No ODP Data

Occasionally search engines will display information from the Open Directory Project in search results instead of using the meta tags on the page. Many webmasters, us included, prefer to prevent this to retain more control over our listings. To do that use the following robots meta tag.

<meta name="robots" content="noodp" />

No Yahoo Directory

This is essentially the same as the NOODP robot tag, except aimed at the Yahoo directory. Our clients usually don’t have as many problems with the Yahoo directory but if you would like to be safe the meta tag is

<meta name="robots" content="noydir" />