One in Four Big Sites Blocks an AI Crawler by Name, but They Block Training Bots Twice as Often as Search Bots: We Read the robots.txt of the Top 1,000 Websites

Disclosure
This article is promotional content produced as part of the Semrush Creator Accelerator programme. It contains affiliate links to Semrush — if you buy through them we may earn a commission at no extra cost to you — and we may also receive a reward from Semrush for publishing it. All measurements in this article are our own. Semrush did not supply, review or approve the data, and the conclusions we draw are ours.
For an AI engine to read your page, a crawler has to be allowed in first. The first place a crawler looks is robots.txt, and many sites have started to name AI bots in it. What is less clear is what they are turning away: the bots that gather training data, or the ones that fetch pages to answer a user's question.
We read the robots.txt files of the 1,000 most popular websites and counted.
Check whether AI bots can reach your pages
Find out if your CDN, firewall or robots.txt blocks AI search bots such as ChatGPT-User, OAI-SearchBot and Perplexity-User with Site Audit in the SEO Toolkit. The AI Visibility Toolkit comes in the same subscription: try Semrush One free for 7 days.
The test
Our population is the 1,000 most popular websites by the Tranco ranking (the 5 November 2026 download), and the request is simply /robots.txt on each, made on 13 November 2026. 510 returned a readable file. 66 had none (a 404), and the rest refused us, timed out or returned a web page instead.
For each file we applied the standard rule: the group that names a bot applies to it, and a bot not named falls under the group for all robots. We counted a bot as blocked by name when its own group says Disallow: / without an Allow: /. We did not count blanket rules for every robot, because those are not a choice about AI: 50 of the 510 files (9.8%) turn away every crawler.
A quarter name an AI bot to block
128 of the 510 sites (25.1%) block at least one AI crawler by name. 95 of them block four or more, 33 block between one and three, and 382 block none.
Training bots are blocked twice as often as search bots
GPTBot, ClaudeBot and CCBot are each blocked by name on 17% to 18% of sites (17.6%, 17.3% and 17.8%). The bots that fetch a page to answer a user are blocked about half as often: OAI-SearchBot on 6.9%, ChatGPT-User on 8.8%, Claude-SearchBot on 8.2%, Claude-User on 8.8% and Perplexity-User on 8.2%. Googlebot (1.0%) and Bingbot (0.8%) are almost never turned away.
The split shows up inside single sites. 55 sites (10.8%) block GPTBot by name but do not block OAI-SearchBot, which is OpenAI's crawler for search, so their pages can still be found and cited while staying out of training.
| Bot | Role | Blocked by name |
|---|---|---|
| GPTBot | OpenAI training | 17.6% (90) |
| ClaudeBot | Anthropic training | 17.3% (88) |
| PerplexityBot | Perplexity search | 12.9% (66) |
| Google-Extended | Google AI control | 14.3% (73) |
| OAI-SearchBot | OpenAI search | 6.9% (35) |
| ChatGPT-User | OpenAI user requests | 8.8% (45) |
| Googlebot | Google Search | 1.0% (5) |
Rank barely matters
The most popular sites are not stricter. GPTBot was blocked by name on 9 of 51 sites in ranks 1 to 100 (17.6%), 30 of 186 in ranks 101 to 500 (16.1%) and 51 of 273 in ranks 501 to 1,000 (18.7%).
What this cannot tell you
- That a bot is actually kept out.
robots.txtis a request that well-behaved crawlers follow. Sites also block at a CDN or firewall, which a text file does not show. - What the other 490 sites do. Their files were unreadable or missing, so nothing here describes them.
- That a block is deliberate. A rule copied from a template counts as a block here.
- That a rule applies to the whole site's value. We looked at whole-site blocks. 61.2% of files restrict some path for GPTBot, such as an admin area, without blocking it.
- That the pattern holds for smaller sites. These are the largest sites, which are more likely to have a policy.
What to do with it
- Read your own
robots.txtas an AI crawler would. List the bots it names and what it says to each. - Decide on training and search separately. Most big sites do: they keep training bots out and leave search bots in.
- Do not block search bots by accident. A rule that targets "all AI" by pattern can shut out the bots that let users find you.
- Check the firewall too. A bot allowed in
robots.txtcan still be blocked by a CDN rule. - Re-check after template changes and after you add a new bot name to a block list.
Check access, track citations, find the pages
Check that AI search bots can reach your pages. Site Audit in the SEO Toolkit reports AI Search Health, including pages blocked from AI search bots.

See which bot is blocked from which page. Filter the Crawled Pages report by a blocked AI bot to list the affected pages.

See whether access turns into citations. Visibility Overview in the AI Visibility Toolkit tracks mentions, citations and cited pages across AI platforms.

The 7-day trial belongs to Semrush One. The AI Visibility Toolkit has none of its own, and Semrush One includes it with the SEO Toolkit. To audit access and track citations without paying first, try Semrush One free for 7 days. Semrush's product page says "Unlimited access to all Semrush One tools" (read on 16 October 2026). That is an offer Semrush publishes; we have not tried it, and no figure in this piece came from a Semrush account.


A policy, not an accident
Most large sites have made a choice about AI bots, and many make it in two parts. Make yours on purpose, and check that the file says what you meant.
Try Semrush One free for 7 days
Site Audit's AI Search Health in the SEO Toolkit and Visibility Overview in the AI Visibility Toolkit, together in Semrush One. Free for 7 days for new users.
Try Semrush One free for 7 daysHow we counted
- Sites: Tranco ranks 1 to 1,000, from the list saved on 5 November 2026. Each domain's
/robots.txtwas fetched once on 13 November 2026, with thewwwform as a fallback. 510 returned plain text and 66 returned 404. - Rules: user-agent groups were read as in the robots exclusion standard. A bot's own group applies if one names it; otherwise the group for
*applies. - Blocked by name: the bot's own group contains
Disallow: /and noAllow: /. A whole-site block that comes only from the*group is counted separately (50 files) and is not included in any bot's figure. - Bots read: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider, Applebot-Extended, meta-externalagent, with Googlebot, Bingbot and Amazonbot for comparison.
- "At least one AI bot": any of the 14 AI bots in that list blocked by name.
- Left out: CDN and firewall rules, and whether any crawler obeys a file.
- Reproduce it:
research/r2-e4-robots-ai-bots/holds the crawler and the analysis script. The raw robots.txt files stay private.
Some links may be affiliate links. We may earn a small commission at no extra cost to you.
Related Articles
Explore more guides and reviews from our experts.

Top Sites With Organization Markup Block AI Crawlers Slightly Less, 18.9% Against 23.1%, but Plain JSON-LD Shows No Such Pattern: Joining Two of Our Studies

We Re-Checked 2,098 URLs That Five AI Engines Cited: 91.6% Still Load, 0.8% Are Gone, and 7.7% Cannot Be Verified by a Bot
