Robots text file is a file which is placed in the root of the website. It decides whether crawlers are permitted or forbidden access to the web site. The file contains a path which should be discovered by security crawlers.