Crawler
A crawler is a program that visits web pages on its own and copies what it finds, so a search engine or AI search can use the contents later.
In a little more detail
Nobody is sitting at a keyboard when this happens. Crawlers identify themselves by name as they arrive, which is what makes it possible to allow some and refuse others. The AI search ones have names like GPTBot, ClaudeBot and PerplexityBot.
Related terms
- robots.txt: robots.txt is a plain text file at the top level of a website that lists which automated visitors are allowed in and which are not.
- Retrieval: Retrieval is the step where AI search fetches live web pages in the middle of answering you, instead of relying only on what it already knows.
See it on your own site
The scan reads your site the way AI search does and reports where you stand on this and sixteen other checks.