Skip to content

One in Four Big Sites Blocks an AI Crawler by Name, but They Block Training Bots Twice as Often as Search Bots: We Read the robots.txt of the Top 1,000 Websites

Sohail Akhtar
Researched by Sohail AkhtarTheToolsVerse
November 15, 20268 min read
AI crawlers in robots.txt on the top 1,000 sites: of 510 readable files, 25.1% block at least one AI bot by name; GPTBot 17.6%, ClaudeBot 17.3%, OAI-SearchBot 6.9%, ChatGPT-User 8.8%, Googlebot 1.0%.

Disclosure

This article is promotional content produced as part of the Semrush Creator Accelerator programme. It contains affiliate links to Semrush — if you buy through them we may earn a commission at no extra cost to you — and we may also receive a reward from Semrush for publishing it. All measurements in this article are our own. Semrush did not supply, review or approve the data, and the conclusions we draw are ours.

For an AI engine to read your page, a crawler has to be allowed in first. The first place a crawler looks is robots.txt, and many sites have started to name AI bots in it. What is less clear is what they are turning away: the bots that gather training data, or the ones that fetch pages to answer a user's question.

We read the robots.txt files of the 1,000 most popular websites and counted.

Check whether AI bots can reach your pages

Find out if your CDN, firewall or robots.txt blocks AI search bots such as ChatGPT-User, OAI-SearchBot and Perplexity-User with Site Audit in the SEO Toolkit. The AI Visibility Toolkit comes in the same subscription: try Semrush One free for 7 days.

The test

Our population is the 1,000 most popular websites by the Tranco ranking (the 5 November 2026 download), and the request is simply /robots.txt on each, made on 13 November 2026. 510 returned a readable file. 66 had none (a 404), and the rest refused us, timed out or returned a web page instead.

For each file we applied the standard rule: the group that names a bot applies to it, and a bot not named falls under the group for all robots. We counted a bot as blocked by name when its own group says Disallow: / without an Allow: /. We did not count blanket rules for every robot, because those are not a choice about AI: 50 of the 510 files (9.8%) turn away every crawler.

A quarter name an AI bot to block

128 of the 510 sites (25.1%) block at least one AI crawler by name. 95 of them block four or more, 33 block between one and three, and 382 block none.

Which crawlers do big sites shut out in robots.txt? Share of the 510 readable robots.txt files among the top 1,000 sites that block each bot by name, 13 November 2026: Bytespider 18.2% (93), CCBot 17.8% (91), GPTBot 17.6% (90), ClaudeBot 17.3% (88), Google-Extended 14.3% (73), Applebot-Extended 13.7% (70), PerplexityBot 12.9% (66), ChatGPT-User 8.8% (45), Claude-User 8.8% (45), Claude-SearchBot 8.2% (42), Perplexity-User 8.2% (42), OAI-SearchBot 6.9% (35), Googlebot 1.0% (5), Bingbot 0.8% (4).
Which crawlers do big sites shut out in robots.txt? Share of the 510 readable robots.txt files among the top 1,000 sites that block each bot by name, 13 November 2026: Bytespider 18.2% (93), CCBot 17.8% (91), GPTBot 17.6% (90), ClaudeBot 17.3% (88), Google-Extended 14.3% (73), Applebot-Extended 13.7% (70), PerplexityBot 12.9% (66), ChatGPT-User 8.8% (45), Claude-User 8.8% (45), Claude-SearchBot 8.2% (42), Perplexity-User 8.2% (42), OAI-SearchBot 6.9% (35), Googlebot 1.0% (5), Bingbot 0.8% (4).

Training bots are blocked twice as often as search bots

GPTBot, ClaudeBot and CCBot are each blocked by name on 17% to 18% of sites (17.6%, 17.3% and 17.8%). The bots that fetch a page to answer a user are blocked about half as often: OAI-SearchBot on 6.9%, ChatGPT-User on 8.8%, Claude-SearchBot on 8.2%, Claude-User on 8.8% and Perplexity-User on 8.2%. Googlebot (1.0%) and Bingbot (0.8%) are almost never turned away.

The split shows up inside single sites. 55 sites (10.8%) block GPTBot by name but do not block OAI-SearchBot, which is OpenAI's crawler for search, so their pages can still be found and cited while staying out of training.

BotRoleBlocked by name
GPTBotOpenAI training17.6% (90)
ClaudeBotAnthropic training17.3% (88)
PerplexityBotPerplexity search12.9% (66)
Google-ExtendedGoogle AI control14.3% (73)
OAI-SearchBotOpenAI search6.9% (35)
ChatGPT-UserOpenAI user requests8.8% (45)
GooglebotGoogle Search1.0% (5)

Rank barely matters

The most popular sites are not stricter. GPTBot was blocked by name on 9 of 51 sites in ranks 1 to 100 (17.6%), 30 of 186 in ranks 101 to 500 (16.1%) and 51 of 273 in ranks 501 to 1,000 (18.7%).

What this cannot tell you

  • That a bot is actually kept out. robots.txt is a request that well-behaved crawlers follow. Sites also block at a CDN or firewall, which a text file does not show.
  • What the other 490 sites do. Their files were unreadable or missing, so nothing here describes them.
  • That a block is deliberate. A rule copied from a template counts as a block here.
  • That a rule applies to the whole site's value. We looked at whole-site blocks. 61.2% of files restrict some path for GPTBot, such as an admin area, without blocking it.
  • That the pattern holds for smaller sites. These are the largest sites, which are more likely to have a policy.

What to do with it

  1. Read your own robots.txt as an AI crawler would. List the bots it names and what it says to each.
  2. Decide on training and search separately. Most big sites do: they keep training bots out and leave search bots in.
  3. Do not block search bots by accident. A rule that targets "all AI" by pattern can shut out the bots that let users find you.
  4. Check the firewall too. A bot allowed in robots.txt can still be blocked by a CDN rule.
  5. Re-check after template changes and after you add a new bot name to a block list.

Check access, track citations, find the pages

Check that AI search bots can reach your pages. Site Audit in the SEO Toolkit reports AI Search Health, including pages blocked from AI search bots.

Semrush's own illustration of the Site Audit overview, from its AI Search Health Knowledge Base page (retrieved 29 October 2026): a demo audit with JavaScript rendering disabled, Site Health 65%, crawled pages by status, an AI Search Health score of 58% and the pages blocked from AI search bots such as ChatGPT-User, OAI-SearchBot and Googlebot. Semrush's illustration, not our account.
Semrush's own illustration of the Site Audit overview, from its AI Search Health Knowledge Base page (retrieved 29 October 2026): a demo audit with JavaScript rendering disabled, Site Health 65%, crawled pages by status, an AI Search Health score of 58% and the pages blocked from AI search bots such as ChatGPT-User, OAI-SearchBot and Googlebot. Semrush's illustration, not our account.

See which bot is blocked from which page. Filter the Crawled Pages report by a blocked AI bot to list the affected pages.

Semrush's own illustration of Site Audit's Crawled Pages report filtered by "Blocked AI search bots: ChatGPT-User", from its Knowledge Base (retrieved 29 October 2026): two pages marked Blocked, each listing the AI agents blocked from it, including ChatGPT-User, OAI-SearchBot and Googlebot. Semrush's illustration, not our account.
Semrush's own illustration of Site Audit's Crawled Pages report filtered by "Blocked AI search bots: ChatGPT-User", from its Knowledge Base (retrieved 29 October 2026): two pages marked Blocked, each listing the AI agents blocked from it, including ChatGPT-User, OAI-SearchBot and Googlebot. Semrush's illustration, not our account.

See whether access turns into citations. Visibility Overview in the AI Visibility Toolkit tracks mentions, citations and cited pages across AI platforms.

Semrush's own illustration of the Visibility Overview report in the AI Visibility Toolkit, from its Knowledge Base (retrieved 9 October 2026): an AI Visibility score of 36/100, an "All AI platforms" selector, and mentions, citations and cited pages over six months. Semrush's illustration, not our account.
Semrush's own illustration of the Visibility Overview report in the AI Visibility Toolkit, from its Knowledge Base (retrieved 9 October 2026): an AI Visibility score of 36/100, an "All AI platforms" selector, and mentions, citations and cited pages over six months. Semrush's illustration, not our account.

The 7-day trial belongs to Semrush One. The AI Visibility Toolkit has none of its own, and Semrush One includes it with the SEO Toolkit. To audit access and track citations without paying first, try Semrush One free for 7 days. Semrush's product page says "Unlimited access to all Semrush One tools" (read on 16 October 2026). That is an offer Semrush publishes; we have not tried it, and no figure in this piece came from a Semrush account.

Semrush One's own product page: "Try free for 7 days" button with "Unlimited access to all Semrush One tools" beneath it. Captured from Semrush's page on 26 September 2026; the same wording was live on 16 October 2026.
Semrush One's own product page: "Try free for 7 days" button with "Unlimited access to all Semrush One tools" beneath it. Captured from Semrush's page on 26 September 2026; the same wording was live on 16 October 2026.
Semrush's official Semrush One graphic, from the Semrush Creator Accelerator affiliate resources: "Semrush One" at the centre of ten capabilities, including AI + SEO Visibility, Site Health + AI Search Auditing, AI Brand Narrative and Share of Voice. Semrush's own image.
Semrush's official Semrush One graphic, from the Semrush Creator Accelerator affiliate resources: "Semrush One" at the centre of ten capabilities, including AI + SEO Visibility, Site Health + AI Search Auditing, AI Brand Narrative and Share of Voice. Semrush's own image.

A policy, not an accident

Most large sites have made a choice about AI bots, and many make it in two parts. Make yours on purpose, and check that the file says what you meant.

Try Semrush One free for 7 days

Site Audit's AI Search Health in the SEO Toolkit and Visibility Overview in the AI Visibility Toolkit, together in Semrush One. Free for 7 days for new users.

Try Semrush One free for 7 days

How we counted

  • Sites: Tranco ranks 1 to 1,000, from the list saved on 5 November 2026. Each domain's /robots.txt was fetched once on 13 November 2026, with the www form as a fallback. 510 returned plain text and 66 returned 404.
  • Rules: user-agent groups were read as in the robots exclusion standard. A bot's own group applies if one names it; otherwise the group for * applies.
  • Blocked by name: the bot's own group contains Disallow: / and no Allow: /. A whole-site block that comes only from the * group is counted separately (50 files) and is not included in any bot's figure.
  • Bots read: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider, Applebot-Extended, meta-externalagent, with Googlebot, Bingbot and Amazonbot for comparison.
  • "At least one AI bot": any of the 14 AI bots in that list blocked by name.
  • Left out: CDN and firewall rules, and whether any crawler obeys a file.
  • Reproduce it: research/r2-e4-robots-ai-bots/ holds the crawler and the analysis script. The raw robots.txt files stay private.

Explore More AI Tools

Browse our curated directory of 855+ verified AI tools.

Browse Directory

Some links may be affiliate links. We may earn a small commission at no extra cost to you.