Why Use Robots.txt file
What Is It?
The purpose of a robots.txt file is to tell search engines not to crawl the contents of specific directories or even individual pages within a website. Since most people welcome Google and all the other search engines to check out their entire website, they don't bother making a robots.txt file. However there are a couple of reasons why all websites would benefit from having one.
The first thing search engines look for is the robots.txt file. If they don't find one they assume it's okay to crawl the entire site and proceed to do so. But looking for a non existent robots file results in a list of "page not found" errors in the web server's error log. By having the robots file, even if it's empty, will keep the error log from getting clogged with those errors thereby making it easier to find broken links and missing files when checking the log. It will also save some bandwith because whenever the server encounters a "page not found" error it consumes bandwith.
How To Make One
Making an empty robots.txt file is simple, just open up a text editor such as notepad, save the blank file as robots.txt or you can add this bit of code before you save it:
user-agent *
Disallow:
That bit of code tells all search engines that they are welcome to crawl all areas of the website, "user-agent" means what search engine bots the instructions are for, the star (*) that comes after it means the instructions are for all bots. "Disallow:" means what parts of the website are not to be crawled, because there is nothing after it, bots know they can crawl everything. Be sure to keep the code on separate lines as in the above example.
And that's all there is to.