Skip to content

Before robots.txt, the Firewall: How 507 Top-Ranking Sites in Five Industries Treat AI Crawlers

Sohail Akhtar
Researched by Sohail AkhtarTheToolsVerse
October 16, 20269 min read
Cover image: 10.1% of 507 sites block at least one AI search bot in robots.txt; 16.2% challenged or refused a plain request to their homepage; 52 of the 53 challenges came from sites behind Cloudflare.

Disclosure

This article is promotional content produced as part of the Semrush Creator Accelerator programme. It contains affiliate links to Semrush — if you buy through them we may earn a commission at no extra cost to you — and we may also receive a reward from Semrush for publishing it. All measurements in this article are our own. Semrush did not supply, review or approve the data, and the conclusions we draw are ours.

An AI answer can only cite a page its crawler could fetch. There are two places a site can say no: its robots.txt, which a polite crawler reads and obeys, and its firewall or CDN, which can refuse the request before any rule is read. In September Cloudflare made the second one more powerful: its announcement says its Block settings "now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot".

Audit your own site for AI crawler blocks

Check which of your pages AI search bots can reach — crawl as OpenAI's search agent with Site Audit in the SEO Toolkit. To get it together with AI visibility tracking, try Semrush One free for 7 days.

We wanted a benchmark, not an anecdote. So we took 100 everyday questions — twenty each in health, personal finance, travel, home & DIY, and science & education — collected Google's US top 10 for every one, and checked the 507 websites that ranked. For each site we read its robots.txt and sent one plain request to its homepage, the kind of request any crawler that does not run JavaScript makes. Here is how the five industries compare.

Who keeps AI crawlers out, by industry: 507 sites in Google's top 10 for 100 everyday questions, US, 16 October 2026. robots.txt blocks at least one AI search bot: home & DIY 21.7% (13 of 60), science & education 17.3%, travel 15.2%, finance 7.2%, health 5.7% (5 of 88). A plain request was challenged or refused: travel 24.1%, home & DIY 22.8%, science & education 14.6%, finance 11%, health 9.9%.
Who keeps AI crawlers out, by industry: 507 sites in Google's top 10 for 100 everyday questions, US, 16 October 2026. robots.txt blocks at least one AI search bot: home & DIY 21.7% (13 of 60), science & education 17.3%, travel 15.2%, finance 7.2%, health 5.7% (5 of 88). A plain request was challenged or refused: travel 24.1%, home & DIY 22.8%, science & education 14.6%, finance 11%, health 9.9%.

The scorecard

IndustrySitesBlock an AI training botBlock an AI search botPlain request turned away
Home & DIY7935.0%21.7%22.8%
Science & education12325.5%17.3%14.6%
Travel11225.3%15.2%24.1%
Finance10913.4%7.2%11.0%
Health11110.2%5.7%9.9%
All 507 sites50718.5%10.1%16.2%

The robots.txt columns are shares of the 395 sites that publish one with rules; the last column is a share of every site. Some sites rank in more than one industry, so the rows add up to more than 507.

Health and finance sites were the most open on both measures. Home & DIY and science & education blocked AI search bots most often in robots.txt; travel and home & DIY turned away the most plain requests.

Which AI crawlers get shut out most

CrawlerWhat it is forSites blocking it in robots.txt (of 395)
Bytespider (ByteDance)training62 (15.7%)
CCBot (Common Crawl)training52 (13.2%)
meta-externalagent (Meta)training48 (12.2%)
ClaudeBot (Anthropic)training46 (11.6%)
Applebot-Extendedtraining43 (10.9%)
Google-Extendedtraining41 (10.4%)
GPTBot (OpenAI)training39 (9.9%)
PerplexityBotsearch36 (9.1%)
Claude-User / Claude-SearchBotsearch28 (7.1%) / 27 (6.8%)
Perplexity-Usersearch26 (6.6%)
ChatGPT-Usersearch25 (6.3%)
OAI-SearchBot (OpenAI)search19 (4.8%)

The difference between the two halves of the table matters. OpenAI's crawler documentation says GPTBot "is used to crawl content that may be used in training our generative AI foundation models", while "OAI-SearchBot is for search." It is also explicit about the cost of blocking the second: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers".

33 sites block training bots and leave search bots alone — the deliberate choice. Every one of the 40 sites that blocks a search bot also blocks training bots: none blocks search alone. That looks less like a decision about AI search and more like a blanket "no AI" list. Among them are some of the biggest sites on the web — Amazon, CNN, The New York Times, Reddit, Pinterest, Quora and Facebook all block AI search crawlers in their robots.txt while ranking in Google's top 10 for our questions.

The layer robots.txt doesn't show

robots.txt is a request. The firewall is a wall. 82 of the 507 sites (16.2%) turned our plain homepage request away: 53 answered with a bot challenge — a "checking your browser" page that only a JavaScript-running browser can pass — and 29 refused outright.

What sits in front of the siteSitesTurned the request away
Cloudflare18629.0% (52 challenges, 2 refusals)
Akamai1931.6% (6 refusals)
Fastly468.7%
Amazon CloudFront567.1%
Vercel1010.0%
Other or none detected1906.8%

52 of the 53 challenges came from sites behind Cloudflare. That is the short version of a problem site owners rarely see: their robots.txt may welcome AI search bots while their CDN's bot protection challenges anything that does not look like a browser.

Two caveats keep this honest. First, our request was not a verified crawler. CDNs keep lists of verified bots that their bot protection can let through, so OpenAI's own crawlers may pass where we did not — but an AI agent that is new, unverified, or fetching on a user's behalf may meet exactly what we met. Second, the managed robots.txt that Cloudflare can write for its customers was rare: its content-signals policy text appeared on 4 of the 507 sites. Most AI blocking we saw was either hand-written robots.txt rules or bot protection at the edge.

Run a five-minute access check on your own site

  1. Open /robots.txt and search for GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot and ClaudeBot. Decide per bot: training or search. Blocking a training bot is not the same as leaving AI answers; blocking OAI-SearchBot is what OpenAI says keeps you out of ChatGPT search answers.
  2. Look for a catch-all. A Disallow: / under User-agent: * blocks every AI crawler you did not allow by name.
  3. Check the firewall, not just the file. In your CDN's bot settings, look for challenge rules, "bot fight" modes and AI-crawler blocks, and confirm that verified search crawlers are allowed.
  4. Crawl your own site as an AI search agent and see which pages come back blocked — every page, not just the homepage.
  5. Watch what happens after you change anything: rankings, AI visibility, and traffic from AI assistants.

Tools for each check

Steps 1 to 4 are a technical audit — the On-Page & Technical SEO side of the SEO Toolkit. Its Site Audit has a Blocked from AI Search report which, in its Knowledge Base's words, "shows how many of the pages checked in your audit are currently blocked from being accessed by AI search engines." It separates the bots that fetch pages "in real time when a user submits a query in an AI assistant" from the rest — the same search-versus-training split as our table.

Semrush's own illustration of Site Audit's Crawled Pages report filtered by "Blocked AI search bots: ChatGPT-User", from its Knowledge Base (retrieved 13 October 2026): two pages marked Blocked, each listing the AI agents blocked from it, including ChatGPT-User, OAI-SearchBot and Googlebot. Semrush's illustration, not our account.
Semrush's own illustration of Site Audit's Crawled Pages report filtered by "Blocked AI search bots: ChatGPT-User", from its Knowledge Base (retrieved 13 October 2026): two pages marked Blocked, each listing the AI agents blocked from it, including ChatGPT-User, OAI-SearchBot and Googlebot. Semrush's illustration, not our account.

The same Knowledge Base page makes the point our firewall table makes: "If you're using a firewall or bot management platform, make sure these user-agent names are also whitelisted there." For step 4, Site Audit's crawler settings list the user agents as "SiteAuditBot (Desktop), SiteAuditBot (Mobile), and OpenAI-Search", and the guide says to "use OpenAI-Search to check whether your site is crawlable by OpenAI's search bot." One more thing to know: Semrush's own crawler has to get past your firewall too — when pages come back inaccessible, its guide says, "your server is likely blocking SiteAuditBot."

Semrush's own illustration of Site Audit's crawler settings, from its AI Search Health Knowledge Base page (retrieved 13 October 2026): a user-agent menu offering SiteAuditBot (Desktop), SiteAuditBot (Mobile) and OpenAI-Search, with crawl-delay options including "Respect robots.txt". Semrush's illustration, not our account.
Semrush's own illustration of Site Audit's crawler settings, from its AI Search Health Knowledge Base page (retrieved 13 October 2026): a user-agent menu offering SiteAuditBot (Desktop), SiteAuditBot (Mobile) and OpenAI-Search, with crawl-delay options including "Respect robots.txt". Semrush's illustration, not our account.

Step 5 has two halves. On the search side, Position Tracking "monitors your website's search rankings daily for specific keywords" and can alert you when they move, while Organic Traffic Insights "allows you to integrate both your Google Analytics and Google Search Console data into one dashboard", which is where you tell a block apart from a change in demand.

Semrush's own illustration of a Position Tracking alert, from its Position Tracking feature page (retrieved 9 October 2026): "Notify me when positions for any keyword for domain.com enter the top 100", with a "Set up alert" button. Semrush's illustration, not our account.
Semrush's own illustration of a Position Tracking alert, from its Position Tracking feature page (retrieved 9 October 2026): "Notify me when positions for any keyword for domain.com enter the top 100", with a "Set up alert" button. Semrush's illustration, not our account.
Semrush's own illustration of the Organic Traffic Insights settings, from its Knowledge Base (retrieved 13 October 2026): a "Connect Your Google account" step linking a Google Analytics account, property and data stream and a Google Search Console property. Semrush's example account, not ours.
Semrush's own illustration of the Organic Traffic Insights settings, from its Knowledge Base (retrieved 13 October 2026): a "Connect Your Google account" step linking a Google Analytics account, property and data stream and a Google Search Console property. Semrush's example account, not ours.

On the AI side, the AI Visibility Toolkit shows whether pages the crawlers can now reach actually get mentioned and cited in AI answers — the outcome the access fix was for.

The SEO Toolkit and the AI Visibility Toolkit can be bought separately; Semrush One includes both, and the 7-day free trial is Semrush One's. If you want both for the five-minute check and the follow-up, try Semrush One free for 7 days — its page offers "Unlimited access to all Semrush One tools", "Offer available to new users only", checked on 14 October 2026. That is Semrush's offer as published, not something we tested; none of the measurements above came from a Semrush account.

Semrush's official Semrush One graphic, from the Semrush Creator Accelerator affiliate resources: "Semrush One" at the centre of ten capabilities — AI + SEO Visibility, Share of Voice, Site Health + AI Search Auditing, Competitor Gap Analysis, Content Optimization, AI Brand Narrative, Automated Reporting, Backlink Analysis & Outreach, Prompt + Keyword Research, and Daily Tracking for Keywords & AI Prompts. Semrush's own image.
Semrush's official Semrush One graphic, from the Semrush Creator Accelerator affiliate resources: "Semrush One" at the centre of ten capabilities — AI + SEO Visibility, Share of Voice, Site Health + AI Search Auditing, Competitor Gap Analysis, Content Optimization, AI Brand Narrative, Automated Reporting, Backlink Analysis & Outreach, Prompt + Keyword Research, and Daily Tracking for Keywords & AI Prompts. Semrush's own image.
Semrush One's own product page: "Try free for 7 days" button with "Unlimited access to all Semrush One tools" beneath it. Captured from Semrush's page on 26 September 2026; the same wording was live on 14 October 2026.
Semrush One's own product page: "Try free for 7 days" button with "Unlimited access to all Semrush One tools" beneath it. Captured from Semrush's page on 26 September 2026; the same wording was live on 14 October 2026.

The takeaway

Most top-ranking sites in our sample are open to AI search: 10.1% block an AI search bot in robots.txt, and in health and finance it is under one in thirteen. The bigger, quieter barrier is at the edge — one site in six turned away a plain request before any robots.txt rule could apply, and almost all of those challenges came from one CDN's bot protection. If you want AI answers to cite you, check the file and the firewall, and decide bot by bot rather than with one "block AI" switch.

Try Semrush One free for 7 days

The SEO Toolkit's Site Audit to find pages AI search bots cannot reach, Position Tracking to catch the drop, and the AI Visibility Toolkit to see the result in AI answers — together in Semrush One. Free for 7 days for new users.

Try Semrush One free for 7 days

Data notes

  1. Sample: Google's US top 10 (desktop, English) for 100 questions, 20 per industry, collected through DataForSEO on 16 October 2026 — 507 distinct websites (subdomains counted separately).
  2. robots.txt: fetched once per site; rules read the way RFC 9309 says a crawler must — the group naming the bot, else the * group, longest match wins. 395 sites published a robots.txt with rules. "Blocks" means the homepage is disallowed for that bot.
  3. Bots: search = OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User; training = GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent, Bytespider, as their operators describe them.
  4. Homepage request: one HTTPS request per site with an ordinary Chrome user agent and no JavaScript, 16 October 2026. "Turned away" = a bot challenge page or a 401/403/406/429/451/503 response; 22 sites that timed out or errored are not counted as blocks. The CDN was read from the response headers.
  5. Sources: Cloudflare, "Accountable mixed-use AI crawlers" (15 September 2026); OpenAI crawler documentation; the Semrush Knowledge Base pages quoted above; screenshots are Semrush's own illustrations, and the official graphic comes from the Creator Accelerator affiliate resources.

Written by Sohail Akhtar. The saved robots.txt files, request results and the analysis script can be re-run.

Explore More AI Tools

Browse our curated directory of 855+ verified AI tools.

Browse Directory

Some links may be affiliate links. We may earn a small commission at no extra cost to you.