It should also be noted that it is optional for search engines to follow the directive of the /roots.txt file. Although all major search engines tend to comply with the /robots.txt file there are plenty of spammy spiders that won’t. For that reason the ony sure fire way to protect content is to place password protection on the sensitive areas.

If you have something that you want to disallow search engine spiders access to, but sill allow unrestricted surfer access the best practice is to create a /robots.txt file in your root directory. That means the url to it should look like this:

http://www.yourwebsite.com/robots.txt

The following are examples of what should be contained in a /robots.txt file:

If you want to exclude ALL robots from your ENTIRE site:

User-agent: *
Disallow: /
If you want to exclude ALL robots from some content on your site:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
If you want to disallow a single robot:

User-agent: EvilRobot
Disallow: /
IF you want to let ONE robot in:

User-agent: GoogleBot
Disallow:
User-agent: *
Disallow: /
The user-agent field is used to identify the bot you would like to allow/deny access. The * character is a wild card character and stands for “any”. Please note that if there is a part of your site that is private it should be password protected even if you use /robots.txt to disallow crawling of the page. /robots.txt is NOT enforceable and although legitimate spiders of search engines will follow its directives, not all bots are benign.

It’s also important to know that just because you have excluded content from being crawled, search engine results can still contain the urls to that content. This is because the spiders may have spotted links to your content from other sites with anchor text suggesting your uncrawled url may be good for that term. Below is a great video from Matt Cutts showing how that works.

Uncrawled URLs in Search Results – Matt Cutts

Best practice is to take those old URL’s, and all their traffic and inbound links, and redirect them to a new working URL. This results in a win/win for your users and your search engine rankings.

That is where the 301 server redirection comes in. Without getting technical a 301 redirect is a HTTP response from the server that means that the page has permanently moved to a new URL. Setting up 301 redirection is pretty straight forward and easy to manage.

301 Redirects and SEO

The permenant redirection indicated by a 301 error is honored by search engines. What this means is that when a search engine encounters a 301 redirect while crawling the web they will delist the old broken URL and update all the incomming links to the new URL.

Here is a short video by Mike Cutts from Google explaining how Google treats 301 redirects and anchor text.

301 Redirects Best Practices

Currently what we suggest to all our clients is that any page that gets search engine traffic or that has a significant number of inbound links get a 301 redirect to a working page with similar or the same content. This provides the best experience for surfers and maximizes the SEO value of their existing inbound links.

Any old pages that don’t get traffic or that have little to no inbound links should then be sent to a well designed and helpful 404 error page. Since there is little value in these pages to be lost the purpose of this method is to maximize the user experience for those who stumble upon one of these pages.

Another popular alternative is to redirect these lesser used pages to the home page. Doing this allows you to direct the inbound links and search engine rankings to a valuable page on the website.

The downside of this method is that potential customers may follow a link expecting to find something and instead find themselves on your home page. This isn’t the best user experience and may result in a lost customer.

Redirects on Apache Servers

The simplist and most common ways to set up 301 redirects is through the .htaccess file in the home directory of your web server. Unlike javascript redirects .htaccess redirects are invisible to the surfer because they are executed by the server when the page is delivered.

.htaccess Single Page Redirects

Single page redirects are best used when the content on the old page is still available at a new location. This way not only do you maintain all the inbound links and pagerank but visitors who arrive through external links find the information they were expecting.

When writing the redirect rule for a single page redirect you leave off the http://www and instead just include the path that would appear afterwords. For exampe to redirect from the old page of http://www.site.com/old/page.html to http://www.site.com/new/page.html you would use …

Redirect 301 old/page.html http://www.site.com/new/page.html

Other 301 Redirects

There are many other ways to set up 301 redirects besides using an .htaccess file. Steven Hargrove has a great resource that covers all of them here. If possible I highly recommend using .htaccess though because of how easy it is to set up and maintain.