Before robots.txt, the Firewall: How 507 Top-Ranking Sites in Five Industries Treat AI Crawlers
Disclosure
This article is promotional content produced as part of the Semrush Creator Accelerator programme. It contains affiliate links to Semrush — if you buy through them we may earn a commission at no extra cost to you — and we may also receive a reward from Semrush for publishing it. All measurements in this article are our own. Semrush did not supply, review or approve the data, and the conclusions we draw are ours.
An AI answer can only cite a page its crawler could fetch. There are two places a site can say no: its robots.txt, which a polite crawler reads and obeys, and its firewall or CDN, which can refuse the request before any rule is read. In September Cloudflare made the second one more powerful: its announcement says its Block settings "now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot".
Audit your own site for AI crawler blocks
Check which of your pages AI search bots can reach — crawl as OpenAI's search agent with Site Audit in the SEO Toolkit. To get it together with AI visibility tracking, try Semrush One free for 7 days.
We wanted a benchmark, not an anecdote. So we took 100 everyday questions — twenty each in health, personal finance, travel, home & DIY, and science & education — collected Google's US top 10 for every one, and checked the 507 websites that ranked. For each site we read its robots.txt and sent one plain request to its homepage, the kind of request any crawler that does not run JavaScript makes. Here is how the five industries compare.
The scorecard
| Industry | Sites | Block an AI training bot | Block an AI search bot | Plain request turned away |
|---|---|---|---|---|
| Home & DIY | 79 | 35.0% | 21.7% | 22.8% |
| Science & education | 123 | 25.5% | 17.3% | 14.6% |
| Travel | 112 | 25.3% | 15.2% | 24.1% |
| Finance | 109 | 13.4% | 7.2% | 11.0% |
| Health | 111 | 10.2% | 5.7% | 9.9% |
| All 507 sites | 507 | 18.5% | 10.1% | 16.2% |
The robots.txt columns are shares of the 395 sites that publish one with rules; the last column is a share of every site. Some sites rank in more than one industry, so the rows add up to more than 507.
Health and finance sites were the most open on both measures. Home & DIY and science & education blocked AI search bots most often in robots.txt; travel and home & DIY turned away the most plain requests.
Which AI crawlers get shut out most
| Crawler | What it is for | Sites blocking it in robots.txt (of 395) |
|---|---|---|
| Bytespider (ByteDance) | training | 62 (15.7%) |
| CCBot (Common Crawl) | training | 52 (13.2%) |
| meta-externalagent (Meta) | training | 48 (12.2%) |
| ClaudeBot (Anthropic) | training | 46 (11.6%) |
| Applebot-Extended | training | 43 (10.9%) |
| Google-Extended | training | 41 (10.4%) |
| GPTBot (OpenAI) | training | 39 (9.9%) |
| PerplexityBot | search | 36 (9.1%) |
| Claude-User / Claude-SearchBot | search | 28 (7.1%) / 27 (6.8%) |
| Perplexity-User | search | 26 (6.6%) |
| ChatGPT-User | search | 25 (6.3%) |
| OAI-SearchBot (OpenAI) | search | 19 (4.8%) |
The difference between the two halves of the table matters. OpenAI's crawler documentation says GPTBot "is used to crawl content that may be used in training our generative AI foundation models", while "OAI-SearchBot is for search." It is also explicit about the cost of blocking the second: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers".
33 sites block training bots and leave search bots alone — the deliberate choice. Every one of the 40 sites that blocks a search bot also blocks training bots: none blocks search alone. That looks less like a decision about AI search and more like a blanket "no AI" list. Among them are some of the biggest sites on the web — Amazon, CNN, The New York Times, Reddit, Pinterest, Quora and Facebook all block AI search crawlers in their robots.txt while ranking in Google's top 10 for our questions.
The layer robots.txt doesn't show
robots.txt is a request. The firewall is a wall. 82 of the 507 sites (16.2%) turned our plain homepage request away: 53 answered with a bot challenge — a "checking your browser" page that only a JavaScript-running browser can pass — and 29 refused outright.
| What sits in front of the site | Sites | Turned the request away |
|---|---|---|
| Cloudflare | 186 | 29.0% (52 challenges, 2 refusals) |
| Akamai | 19 | 31.6% (6 refusals) |
| Fastly | 46 | 8.7% |
| Amazon CloudFront | 56 | 7.1% |
| Vercel | 10 | 10.0% |
| Other or none detected | 190 | 6.8% |
52 of the 53 challenges came from sites behind Cloudflare. That is the short version of a problem site owners rarely see: their robots.txt may welcome AI search bots while their CDN's bot protection challenges anything that does not look like a browser.
Two caveats keep this honest. First, our request was not a verified crawler. CDNs keep lists of verified bots that their bot protection can let through, so OpenAI's own crawlers may pass where we did not — but an AI agent that is new, unverified, or fetching on a user's behalf may meet exactly what we met. Second, the managed robots.txt that Cloudflare can write for its customers was rare: its content-signals policy text appeared on 4 of the 507 sites. Most AI blocking we saw was either hand-written robots.txt rules or bot protection at the edge.
Run a five-minute access check on your own site
- Open
/robots.txtand search for GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot and ClaudeBot. Decide per bot: training or search. Blocking a training bot is not the same as leaving AI answers; blocking OAI-SearchBot is what OpenAI says keeps you out of ChatGPT search answers. - Look for a catch-all. A
Disallow: /underUser-agent: *blocks every AI crawler you did not allow by name. - Check the firewall, not just the file. In your CDN's bot settings, look for challenge rules, "bot fight" modes and AI-crawler blocks, and confirm that verified search crawlers are allowed.
- Crawl your own site as an AI search agent and see which pages come back blocked — every page, not just the homepage.
- Watch what happens after you change anything: rankings, AI visibility, and traffic from AI assistants.
Tools for each check
Steps 1 to 4 are a technical audit — the On-Page & Technical SEO side of the SEO Toolkit. Its Site Audit has a Blocked from AI Search report which, in its Knowledge Base's words, "shows how many of the pages checked in your audit are currently blocked from being accessed by AI search engines." It separates the bots that fetch pages "in real time when a user submits a query in an AI assistant" from the rest — the same search-versus-training split as our table.

The same Knowledge Base page makes the point our firewall table makes: "If you're using a firewall or bot management platform, make sure these user-agent names are also whitelisted there." For step 4, Site Audit's crawler settings list the user agents as "SiteAuditBot (Desktop), SiteAuditBot (Mobile), and OpenAI-Search", and the guide says to "use OpenAI-Search to check whether your site is crawlable by OpenAI's search bot." One more thing to know: Semrush's own crawler has to get past your firewall too — when pages come back inaccessible, its guide says, "your server is likely blocking SiteAuditBot."

Step 5 has two halves. On the search side, Position Tracking "monitors your website's search rankings daily for specific keywords" and can alert you when they move, while Organic Traffic Insights "allows you to integrate both your Google Analytics and Google Search Console data into one dashboard", which is where you tell a block apart from a change in demand.


On the AI side, the AI Visibility Toolkit shows whether pages the crawlers can now reach actually get mentioned and cited in AI answers — the outcome the access fix was for.
The SEO Toolkit and the AI Visibility Toolkit can be bought separately; Semrush One includes both, and the 7-day free trial is Semrush One's. If you want both for the five-minute check and the follow-up, try Semrush One free for 7 days — its page offers "Unlimited access to all Semrush One tools", "Offer available to new users only", checked on 14 October 2026. That is Semrush's offer as published, not something we tested; none of the measurements above came from a Semrush account.


The takeaway
Most top-ranking sites in our sample are open to AI search: 10.1% block an AI search bot in robots.txt, and in health and finance it is under one in thirteen. The bigger, quieter barrier is at the edge — one site in six turned away a plain request before any robots.txt rule could apply, and almost all of those challenges came from one CDN's bot protection. If you want AI answers to cite you, check the file and the firewall, and decide bot by bot rather than with one "block AI" switch.
Try Semrush One free for 7 days
The SEO Toolkit's Site Audit to find pages AI search bots cannot reach, Position Tracking to catch the drop, and the AI Visibility Toolkit to see the result in AI answers — together in Semrush One. Free for 7 days for new users.
Try Semrush One free for 7 daysData notes
- Sample: Google's US top 10 (desktop, English) for 100 questions, 20 per industry, collected through DataForSEO on 16 October 2026 — 507 distinct websites (subdomains counted separately).
- robots.txt: fetched once per site; rules read the way RFC 9309 says a crawler must — the group naming the bot, else the
*group, longest match wins. 395 sites published a robots.txt with rules. "Blocks" means the homepage is disallowed for that bot. - Bots: search = OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User; training = GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent, Bytespider, as their operators describe them.
- Homepage request: one HTTPS request per site with an ordinary Chrome user agent and no JavaScript, 16 October 2026. "Turned away" = a bot challenge page or a 401/403/406/429/451/503 response; 22 sites that timed out or errored are not counted as blocks. The CDN was read from the response headers.
- Sources: Cloudflare, "Accountable mixed-use AI crawlers" (15 September 2026); OpenAI crawler documentation; the Semrush Knowledge Base pages quoted above; screenshots are Semrush's own illustrations, and the official graphic comes from the Creator Accelerator affiliate resources.
Written by Sohail Akhtar. The saved robots.txt files, request results and the analysis script can be re-run.
Some links may be affiliate links. We may earn a small commission at no extra cost to you.
Related Articles
Explore more guides and reviews from our experts.

Top Sites With Organization Markup Block AI Crawlers Slightly Less, 18.9% Against 23.1%, but Plain JSON-LD Shows No Such Pattern: Joining Two of Our Studies

We Re-Checked 2,098 URLs That Five AI Engines Cited: 91.6% Still Load, 0.8% Are Gone, and 7.7% Cannot Be Verified by a Bot
