Robots.txt

Robots.txt

What is robots.txt?

The robots.txt is a file that helps to prevent web crawlers, search engine robots, and other automated tools from accessing specific webpages or files on your site. It can also provide instructions to these tools on how they should interact with your site, how often they should check for updates, which files or pages to index, and which areas are off-limits. By having a robots.txt, you can make sure your website’s data is safe and protected, and that you have full control over who has access to your webpages. This might seem contrary to the goals of SEO, but preventing spiders from exploring some of your content in certain cases can be of benefit, such as if you have duplicate content on your website and do not want to be penalized or you are building a new site. 

Common mistakes with robots.txt

Robots.txt tells search engine spiders not to crawl certain pages, and some site owners mistakenly think that this is a good way to keep certain information private. There are plenty of malicious spiders that will not respect the protocol, and steal your information. If you have personal information on your website, you need much stronger security. 

Where should robots.txt be located?

In the root directory of your website or URL. 

When using robots.txt, it is a good idea to be familiar with the * symbol. A * tells the spider that this command applies to all web crawlers. A / will indicate that the rule applies to all the pages on a particular site. 

When used wisely, robots.txt can be a valuable resource and help you control what and where search engine spiders can explore your offerings. 

Related: Google’s URL Inspection Tool