Robots.txt is a text file in the root directory of a web directory and contains instructions to user agents and crawlers on how to handle the website's content. In other words: may specific directories be indexed, which pages should be kept out of the search engine index? The main commands are: "user-agent," "allow," and "disallow." 

The robot file can be checked via the browser by adding /robots.txt to the currently valid domain. 

Example: https://www.yourwebsite.com/robots.txt 

Is the displayed file empty, or is the server sending an error message? Then there is a need for action! 

Note: In the competitive analysis, it can be helpful to take a look at the Robots.txt entries of the competitors. This may result in approaches for your projects, especially for larger eCommerce projects or extensive websites. 

Purpose and format of the Robots.txt file 

The text file has two specific tasks: 

Keep unwanted / duplicate / empty pages out of the index and 

affect the crawl budget per website. 

This purpose was first defined in 1994 and later expanded further. Since then, it has been used extensively. However, the Robots Exclusion Protocol (REP) is not one of the official Internet standards. Search engines usually observe the crawl instructions in Robots.txt, but they are free to implement and are quite able to carry out crawl processes that differ from them. As a user agent, Google usually adheres to the specifications of the file, but it looks different from some other user agents. 

Common CMS usually creates a robots.txt file during the basic configuration. If the file is missing after a relaunch or CMS change, it can be created empty with a simple text editor and uploaded to the website's root directory. 

The content can be edited and supplemented at any time and should be checked regularly for necessary adjustments. 

Google provides a test tool in which changes to the file can be checked before implementation. But Bing also has a Robots.txt tester. 

Important: The tool only checks the entries for the GoogleBot user agent and the services connected to GoogleBot, i.e.: 

  • GoogleBot image 
  • GoogleBot news 
  • GoogleBot video 
  • Media partners-Google 
  • GoogleBot Mobile 
  • Adsbot-Google