The ten second check
Open your own robots.txtrobots.txtrobots.txt is a plain text file at the top level of a website that lists which automated visitors are allowed in and which are not.. Type your website address into a browser and add /robots.txt to the end, so it looks like yoursite.com/robots.txt. You are looking for a block of text that mentions Cloudflare, usually written as a comment line beginning #, followed by rules like this:
# BEGIN Cloudflare Managed content User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: /
If that is there, Cloudflare is turning away AI searchAI searchAI search is a program you ask questions in plain language, like ChatGPT, Claude, Gemini or Perplexity, which writes you an answer instead of handing you a list of links., and nothing else you do to your site will fix it.
Why your own file says something different
This is the part that costs people months. The robots.txt in your project can say every crawlerCrawlerA crawler is a program that visits web pages on its own and copies what it finds, so a search engine or AI search can use the contents later. is welcome, and be completely correct, and be irrelevant, because the visitor never reaches your server to read it.
Cloudflare sits in front of your website. Every request arrives there first. When the managed setting is on, Cloudflare answers the request for robots.txt itself, with its own file, and your version is never consulted.
So the usual debugging fails. You check the file, the file is right, you assume the problem is elsewhere, and you go looking in the wrong place for weeks.
Who it affects
Two groups, and the second is much larger than people realize.
- Anyone who switched it on. The setting is a toggle in the Cloudflare dashboard called AI Scrapers and Crawlers. It was widely recommended when it launched, often by people who were thinking about content theft rather than about being found.
- Anyone who never touched it. Cloudflare announced that from 1 July 2025, new domains on its network would block these crawlers by default. If your site was set up after that and nobody deliberately turned this off, the odds are that it is on.
The crawlers it turns away are the ones that matter: GPTBotCrawlerA crawler is a program that visits web pages on its own and copies what it finds, so a search engine or AI search can use the contents later. and ChatGPT-User from OpenAI, ClaudeBot from Anthropic, PerplexityBot, and Google-Extended, which governs Google’s AI features. These are the ones that do the retrievalRetrievalRetrieval is the step where AI search fetches live web pages in the middle of answering you, instead of relying only on what it already knows. when somebody asks a question.
Should you turn it off?
Genuinely not obvious, and the two sides both have a case.
- Leave it on if your content is the product and you do not want it absorbed into the training dataTraining dataTraining data is the enormous body of text AI search learned from before it was released, which is fixed, often months or years old, and cannot be edited by you. of models you have no relationship with. Publishers, course makers, research outfits and anyone with a paywall have real reasons, and losing AI visibility is a price they may be happy to pay.
- Turn it off if you are a business that wants to be recommended. You are not protecting a library, you are hiding a shop. Every buyer who asks AI search for a recommendation in your category is being answered without you in the sentence.
What is not defensible is having it on without knowing. That is the situation this page exists for.
How to turn it off
In the Cloudflare dashboard: Security, then Bots, then switch off AI Scrapers and Crawlers. Then reload yoursite.com/robots.txt and confirm the managed block is gone.
Two things to expect afterwards. The file changes immediately, but being read again does not: crawlers return on their own schedule, so give it days rather than minutes. And turning it off restores access, it does not create demand. If AI search was not naming you for other reasons as well, removing the block reveals that rather than fixing it. Your mention rateMention rateMention rate is the share of answers that named your business, out of all the times a question was asked. is what tells you which of the two you are looking at.
How we check it
Our scan reads the robots.txt that is actually being served, which is the same one AI search gets, not the one in your repository. When it finds AI search crawlers blocked, it looks at whether the rules were injected by Cloudflare, and if they were, it says so and names the setting instead of leaving you to find it.
The long version
What is GEO, and how do you actually do it covers all of this from the beginning, including the parts this page skipped.