robots.txt
robots.txt is a plain text file at the top level of a website that lists which automated visitors are allowed in and which are not.
In a little more detail
You can read any site's copy by adding /robots.txt to the end of its address, including your own. It is the most common single cause of a business being invisible to AI search, usually because a rule written years ago to keep scrapers out now also turns away the AI search nobody had heard of at the time. Saying nothing about a visitor is generally the same as allowing it.
Related terms
- Crawler: A crawler is a program that visits web pages on its own and copies what it finds, so a search engine or AI search can use the contents later.
- llms.txt: llms.txt is an optional plain text file at the top level of a website that tells AI search, in ordinary language, what the site is and which pages matter most.
See it on your own site
The scan reads your site the way AI search does and reports where you stand on this and sixteen other checks.