We Tested 358 Sites That AI Engines Cite for Crawler Blocks. The Usual Test Gets It Wrong.

On 15 September 2026 Cloudflare changed how it treats AI crawlers. In its announcement it says its Block settings "now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot" — so blocking AI training that way can stop a search crawler too. The same post gives the scale of the decision site owners are making: 17% of Cloudflare sites use some mechanism to block AI training, and less than 1% block search bots.

That makes a practical question urgent: can the AI crawlers you care about actually read your pages? The test most people reach for is a request that announces itself as GPTBot, or ClaudeBot, or Googlebot, to see what comes back. We ran that test on 358 websites at once — every site ChatGPT, Gemini or Google AI Mode cited in our 40-question panel — alongside their robots.txt files. Then we checked a week of our own Cloudflare logs to see which of the "bots" that reach us are real.

Few of the sites AI engines cite block AI crawlers on paper: 12 of 256 with a robots.txt disallow GPTBot. But the request test reported blocks that cannot be real — 8 sites refused a request calling itself Googlebot, including Wikipedia, Adobe and Zendesk. And on our own site, only 5.5% of requests naming GPTBot were actually GPTBot.

How we measured

• Sites: the 358 distinct websites cited by ChatGPT, Gemini or Google AI Mode for 40 questions about AI software on 9 October 2026 — vendors, publishers, directories, forums, Google and OpenAI properties.
• robots.txt: fetched once per site and parsed the way the standard (RFC 9309) says crawlers should: the group naming the crawler wins over *, the longest matching rule wins, and Allow beats Disallow on a tie. We recorded whether each of 15 crawlers may fetch the homepage.
• Request test: one request to each homepage as a normal Chrome browser, then one each with the published user-agent strings of Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot. A request was refused if it got 401, 403, 406, 429, 451 or 503, or a Cloudflare challenge. Requests to each site were spaced out and sent one at a time, on 13 October 2026 between 13:06 and 19:31 UTC.
• Our own logs: Cloudflare's analytics for thetoolsverse.com, 6–13 October, split by whether Cloudflare verified each request as the bot its user agent named.

250 of the 358 sites served our browser request normally; the request-test figures use those 250 as the base. 36 refused even a browser (they challenge every automated client), and 70 timed out or failed to connect.

What robots.txt says

Base: 256 sites with a readable robots.txt.

The sites AI engines cite are, with few exceptions, open to AI crawlers. Many name GPTBot or ClaudeBot in their robots.txt — and then allow them. Only 18 of 256 disallow any of the seven AI crawlers from OpenAI, Anthropic, Perplexity, Google and Common Crawl at the homepage, and the crawler most often refused was ByteDance's, not any of theirs. That is unsurprising: a site that shuts AI crawlers out is less likely to be cited in the first place.

The exceptions are instructive, because they show that the crawler names mean different things. OpenAI's crawler documentation says GPTBot "is used to crawl content that may be used in training our generative AI foundation models", while "OAI-SearchBot is for search." So we checked every page ChatGPT cited in three runs of our panel against its own site's robots.txt, path by path. Of 314 cited pages we could check, 9 were disallowed to GPTBot — seven of them on canva.com — and ChatGPT cited them anyway. Blocking the training crawler did not keep those pages out of ChatGPT's answers.

Only one cited page was disallowed to OpenAI's search agents: claude.ai's homepage. OpenAI's documentation covers that case too: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers", though they "can still appear as navigational links."

What a request that calls itself a bot gets

Read at face value, a tenth of the sites AI engines cite block GPTBot. The rest of the table says not to read it at face value.

Nine sites refused a request claiming to be Googlebot or Bingbot — among them en.wikipedia.org, adobe.com, zendesk.com, reddit.com and code.visualstudio.com. All five of those named here are in Google's index and refused our "Googlebot" request. They are not blocking Googlebot; they are blocking a request that says it is Googlebot and comes from somewhere Googlebot does not. Search engines and AI companies publish the addresses their crawlers use — OpenAI lists separate IP ranges for GPTBot, OAI-SearchBot and ChatGPT-User — so a firewall can check the claim, and a request from an ordinary machine fails it.

The clearest case is OpenAI itself. openai.com and chatgpt.com refused our requests calling themselves GPTBot, OAI-SearchBot and ChatGPT-User, and help.openai.com challenged the OAI-SearchBot and ChatGPT-User ones — yet ChatGPT cited help.openai.com in our panel. Twelve sites that ChatGPT cited refused at least one of our "OpenAI" requests.

And the two layers disagree with each other. Of the 21 sites that refused our GPTBot request and have a readable robots.txt, 18 allow GPTBot in that file. Of the 9 that disallow it in robots.txt and served our browser, 6 served our GPTBot request without complaint — robots.txt is a request to crawlers, not a lock.

So the curl test fails in both directions. It reports blocks that only apply to impersonators, as the Googlebot rows show. And it cannot see blocks that only apply to verified bots, because a test request is never verified. That second blind spot is the one Cloudflare's September change makes more likely: a rule aimed at verified crawlers is invisible to any test you run from your own laptop.

Of the 27 sites that refused at least one AI user agent, 17 were on Cloudflare, which served 160 of the 358 sites in total.

Our own site: who was really knocking

From the other side of the door, the impersonation is easy to see. Over the seven days to 13 October, Cloudflare logged every request to our site whose user agent named one of these bots, and marked which ones it could verify as that bot.

• 345 requests said they were GPTBot. Cloudflare verified 19. The other 326 were something else.
• 583 of 1,459 "Googlebot" requests were not Googlebot — 40%.
• Bingbot was almost always real: 10,275 of 10,362.
• No PerplexityBot request could be verified, because Cloudflare de-listed Perplexity as a verified bot in August 2025 after finding undeclared crawling. We served 1,677 of those 2,123 requests normally; we cannot tell from our side which were Perplexity's.

We could see what the impersonators wanted. The fake GPTBot and Googlebot requests that our firewall refused were asking for paths like /.env, /admin%2F.env, /.htpasswd, /rclone.conf, /config/env/aws_credentials.env and /.git/config — credential files, not content. They were vulnerability scanners wearing a crawler's name, blocked by custom rules we wrote to stop scanners. Those rules exempt bots Cloudflare has verified, and the result shows in the logs: verified Googlebot, Bingbot, GPTBot, ChatGPT-User and ClaudeBot got no 403s at all in the week. Verified OAI-SearchBot got 50, every one on a path that does not exist on our site — 49 of them .php files on a site with no PHP.

That is the asymmetry the request test misses. Our firewall refuses requests that say "GPTBot". It serves GPTBot. A curl test against our site would have reported us as blocking AI crawlers.

What this means in practice

• Read robots.txt as policy, not as reality. It tells you what you asked crawlers to do. Whether they can reach you is decided elsewhere — in a firewall, a CDN setting or a bot-management default.
• Know which crawler does what. On OpenAI's own definitions, GPTBot is for training and OAI-SearchBot is for search. Blocking the first does not, on our data, keep you out of ChatGPT's answers. Blocking the second is what does.
• Do not trust a refused request from your own machine. If a site refuses a "Googlebot" request while ranking in Google, the test is wrong, not the site. Run the request test to find rules keyed to user agents, then confirm with your logs.
• Check the verified-bot view in your CDN. The question that matters is whether verified crawlers got 200s on real pages. On Cloudflare, that is the verified-bot split of your traffic analytics.
• Re-check after any infrastructure change. Cloudflare's September change applied defaults to new domains and changed what the Block settings do. A setting nobody touched can still mean something different this month.

What we could not test

• What the real bots receive on other people's sites. We can only see verified traffic for our own domain. For the 358 sites, we know their stated policy and how they treat an unverified claim — not what GPTBot itself gets.
• Deep pages. The request test hit homepages; a site can treat its blog differently. The path-level robots.txt check covered every cited page, but not the firewall.
• Sites that failed to answer. 70 timed out or refused connections even to a browser-like request, and 36 challenged every automated request. We did not try to get past either.
• One week of logs. Seven days of one site's traffic, from Cloudflare's sampled analytics.

Run this on your own site

1. Read your robots.txt as a crawler would — for each AI crawler by name, can it fetch your key templates? Remember that the most specific group wins.
2. Check the bot settings in your CDN or firewall, especially after Cloudflare's 15 September changes. Write down what Search, Training and Agent are set to.
3. Open the verified-bot view of your traffic and look for 401s, 403s and challenges served to verified crawlers on real pages. That is a real block.
4. Treat requests that only claim a bot's name as untrusted. Look at what they request. Ours wanted credentials.
5. Crawl the whole site, not just the homepage, for robots and noindex rules that stop AI agents, and repeat it on a schedule.

Where Semrush fits

Step five is a crawler's job, and on the technical side it is what Semrush's On-Page & Technical SEO tools are for. Site Audit has a Blocked from AI Search report which, in its Knowledge Base's words, "shows how many of the pages checked in your audit are currently blocked from being accessed by AI search engines." It checks eight agents — ChatGPT-User, OAI-SearchBot, Googlebot, Google-Extended, Perplexity-User, PerplexityBot, Claude-User and Claude-SearchBot — and separates the bots that fetch pages "in real time when a user submits a query in an AI assistant" from those that "scrape content at scale for training large language models." That is the split our robots.txt table turns on — blocking a training crawler is not blocking an assistant or search crawler — applied to every page instead of a homepage.

The same Knowledge Base page adds the caveat our data supports: "If you're using a firewall or bot management platform, make sure these user-agent names are also whitelisted there." robots.txt and the firewall are separate layers, and an audit of one does not clear the other.

Site Audit can also crawl as OpenAI's search agent. Its configuration guide lists the user-agent options as "SiteAuditBot (Desktop), SiteAuditBot (Mobile), and OpenAI-Search", and says to "use OpenAI-Search to check whether your site is crawlable by OpenAI's search bot." Read alongside our results, that is the request test at scale — every page, on a schedule. Like any request test, it shows rules keyed to the user agent, so pair it with your CDN's verified-bot view, which is the only place a block aimed at verified crawlers shows up.

A block that slips through shows up first as a drop. Position Tracking checks a fixed keyword list daily — its Knowledge Base describes it as a tool that "monitors your website's search rankings daily for specific keywords".

To tell a block from a change in demand, Organic Traffic Insights "allows you to integrate both your Google Analytics and Google Search Console data into one dashboard", which is where you separate "we got blocked" from "demand moved". Once access is confirmed, the AI Visibility Toolkit answers the next question — whether reachable pages are actually mentioned and cited.

The audit, the tracking and the AI visibility checks above all sit in Semrush One, which Semrush's own affiliate graphic sums up as a set of linked capabilities — Site Health + AI Search Auditing and AI + SEO Visibility among them.

Two planning notes. The free trial belongs to Semrush One, not to a single toolkit: Semrush One's page offers "Try free for 7 days" with "Unlimited access to all Semrush One tools" — "Offer available to new users only", checked on 13 October 2026. And Semrush's own crawler has to get through your firewall to audit you; its guide says that when pages come back inaccessible, "your server is likely blocking SiteAuditBot." Every measurement in this article was made without a Semrush account.

What we actually learned

The sites AI engines cite are overwhelmingly open to AI crawlers: 12 of 256 disallow GPTBot, and blocking GPTBot did not stop ChatGPT citing nine of their pages. The test people use to check access is less reliable than the policy it checks. Wikipedia refused it as a fake Googlebot, openai.com refused it as fake OpenAI crawlers, and it would have reported our site — which refused no verified AI crawler on any real page all week — as blocking GPTBot.

Access is decided by identity, not by name. Check what you declare in robots.txt, check what your CDN does to verified crawlers, and treat any request that only claims to be a bot as exactly that: a claim.

Sources and method

Written by Sohail Akhtar. Every figure here is a measurement on a stated date; the raw responses for all 358 sites, the robots.txt parser and the Cloudflare export are saved and can be re-run.