User agent
A user agent is the name a visitor gives when it arrives at your site, which is how you can tell a browser from a crawler.
In a little more detail
Rules in robots.txt are written against these names. It is also how a site can be told to serve one thing to people and another to machines, which is a bad idea and detectable.
Related terms
- Crawler: A crawler is a program that visits web pages on its own and copies what it finds, so a search engine or AI search can use the contents later.
- robots.txt: robots.txt is a plain text file at the top level of a website that lists which automated visitors are allowed in and which are not.
See it on your own site
The scan reads your site the way AI search does and reports where you stand on this and sixteen other checks.