How to stop search engines from indexing your pages

Kate Bolin | 12 July 2006

Got something you want to put online, but you don't really want it showing up in search engine results? Here are two quick and easy solutions.

Use a specific meta tag

For each page you don't want to appear in search engine results, have only one <meta> tag. Not a description, not some keywords, just a single <meta> tag for robots.

<meta name="robots" content="noindex,nofollow,noarchive" />

Pop that into the <head> of each page, and you're telling search engines not to index the page, not to follow any links from the page, and not to archive the page.

Create a robots.txt file

If your pages are all in a separate directory, you can also block search engines by using a robots.txt file.

Create a text file and, in it, disallow all the directories you want protected:

User-agent: *
Disallow: /nameofdirectory
Disallow: /anothernameofdirectory

Do it for all the directories you want, then save the file as robots.txt, and upload it to your main directory. The search engine robots will hit the robots.txt, find out which directories you don't want them sniffing in, and skip them.

So there you go. Two little things that can save you a world of trouble.

However, these aren't completely effective solutions. If you really want to block search engines from accessing your pages, you can either password-protect your pages, or keep them offline.

The choice is yours. Have fun!

Permanent link | Filed under: Search engine optimization, General interest

About the Author

Kate Bolin alternates between web design and content production in Nottingham, England, and currently works as the web person for a small NHS project.

When she’s not wrangling XHTML and CSS or fighting for truth, justice, and accessibility, she gets random on Katemonkey.co.uk and attends Multipack meetings. She likes monkeys, sleep, and the little toys you get in Kinder Suprise.

Articles by Kate Bolin

5 Comments

| Post a comment | Comments Feed

Rik Lomas : http://rikrikrik.com

12 July 2006, 12:10 : Permanent link to comment

Or… just build it in Flash. Then no one will find your information…
Nick Day : http://www.njday.com

12 July 2006, 12:21 : Permanent link to comment

Shouldn’t there be a trailing ”/” after the directory names?

According to Wikipedia :

The robots.txt patterns are matched by simple substring comparisons, so care should be taken to make sure that patterns matching directories have the final ’/’ character appended: otherwise all files with names starting with that substring will match, rather than just those in the directory intended.
Kate Bolin : http://www.katemonkey.co.uk/

12 July 2006, 13:22 : Permanent link to comment

Shouldn’t there be a trailing ”/” after the directory names?

Yeah, Nick, having the ”/” is the most effective way to do it. I should edit the article, but then it’d make the comments silly.

I, for my sins, forget to do that fairly often, just because it’s rare I have files named the same.
reese : http://designbyreese.com

13 July 2006, 06:10 : Permanent link to comment

Aware that people can sometimes read my robots file, and I have something to hide not just from search engines but from people poking around my files, I will do weird, unique names but not fully type them out. Example:

directory called chamelion, which is a software system folder. I might put

/chame
in my robots

So someone can’t figure out the full name and try to call the directory accordingly.
Sam : http://www.r3new.com

18 July 2006, 06:52 : Permanent link to comment

Wouldn’t using content="none" have the same effect with less code?

Commenting is closed for this article.