Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

Information About Robots.txt

What is robots.txt?

Robots.txt is a file that lives on your website at examplesite.com/robots.txt that specifies which content you want inside search engines. This means that any content that you do not want to be indexed cannot be crawled or found through search (in most cases). The reason we say "in most cases" is because there is some software that can ignore your Robots.txt file, which will allow that software to crawl your whole site.

More information on robots.txt.