AI crawler
An AI crawler is the specific automated visitor an AI search company sends to read your pages, such as GPTBot, ClaudeBot or PerplexityBot.
In a little more detail
They are separate from search engine crawlers and can be allowed or blocked independently. Blocking Googlebot and blocking GPTBot are two different decisions with two different costs.
Related terms
- Crawler: A crawler is a program that visits web pages on its own and copies what it finds, so a search engine or AI search can use the contents later.
- robots.txt: robots.txt is a plain text file at the top level of a website that lists which automated visitors are allowed in and which are not.
See it on your own site
The scan reads your site the way AI search does and reports where you stand on this and sixteen other checks.