- Read your robots.txt before changing anything. On Shopify it lives at /robots.txt and is generated from a template you can edit through robots.txt.liquid.
- Decide training and retrieval separately. If you object to training, block GPTBot and Google-Extended and leave OAI-SearchBot and ChatGPT-User allowed.
- A blanket 'User-agent: * / Disallow: /' block will catch every agent, including the ones that would have sent you buyers.
Why an agent cares
Most robots.txt files that block AI crawlers were not written as a considered decision. They were copied from a template, added by a plugin during the 2023 scraping panic, or inherited from an agency that set the store up years ago.
The trade has changed since then. Blocking these crawlers used to mean protecting your copy from being used as training data. Today it also means removing yourself from the retrieval index that answers shopping questions. Those are two different systems, and the same directive turns off both.
It is worth being precise about which you actually object to. Training and retrieval are separable for some crawlers: OpenAI splits GPTBot (training) from OAI-SearchBot (retrieval), and Google splits Google-Extended (training) from ordinary Googlebot. You can decline to feed the model and still be findable.
Allow retrieval, decline training
# Declines training
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Allows shopping answers to cite you
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /Check your own store
This check runs as part of a full audit, alongside 18 others. Nothing to install.