New Hampshire search engine optimization

(Robots.txt)

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.Robots.txt Generatortool is used to generate a simple robots.txt file for your website.This generator tool is designed to create the text for the file for you; you can then make changes afterward should you deem it necessary.Generating a robot file with this tool is ideal if you wish to block certain directories or files from search engines.To use the generator tool, enter the required information below and click the button. You will then be shown the text for the file. Copy this to a file called robots.txt and place on the root of your website .The location of robots.txt is very important. It must be in the main directory because otherwise user agents will not be able to find it – they do not search the whole site for a file named robots.txt. Instead, they look first in the main directory and if they don't find it there, they simply assume that this site does not have a robots.txt file and therefore they index everything they find along the way.

References- http://www.seochat.com/seo-tools/robots-generator/

                     http://www.businessseek.biz/article-directory/article-59.html

                     http://www.webconfs.com/what-is-robots-txt-article-12.php