“I Robots.txt” - aka: “How To Control Search Engines”
Wednesday, November 7th, 2007You may think that you have no control over what the search engines see or do with you site…. but you’re wrong mon ami. Using a simple file created with Windows Notepad (which every PC has), you can tell the search engines - including Google - what to look at and what not to look at when they visit your website.
It’s done by directing the search engine robots with a robots.txt file. With a few easy to implement lines of controls, you can hide duplicate content, admin areas, private folders, confidential pages, and more from the search engines.
(more…)