Skip to content

Top Sites With Organization Markup Block AI Crawlers Slightly Less, 18.9% Against 23.1%, but Plain JSON-LD Shows No Such Pattern: Joining Two of Our Studies

Sohail Akhtar
Researched by Sohail AkhtarTheToolsVerse
November 15, 20267 min read
Share of 437 top sites that block at least one AI bot in robots.txt: Organization markup 18.9% against 23.1% without; sameAs links 18.2% against 23.1%; any JSON-LD 22.7% against 20.1%.

Disclosure

This article is promotional content produced as part of the Semrush Creator Accelerator programme. It contains affiliate links to Semrush — if you buy through them we may earn a commission at no extra cost to you — and we may also receive a reward from Semrush for publishing it. All measurements in this article are our own. Semrush did not supply, review or approve the data, and the conclusions we draw are ours.

Two pieces of advice often sit side by side. Describe your organisation in structured data so machines can understand who you are. And decide which AI crawlers may read your pages at all. They are usually discussed separately, yet they seem to pull in one direction: a site that labels itself for machines presumably wants machines to read it.

Does that show up in practice? We had both measurements for the same websites, so we joined them.

See whether AI platforms use your pages

Follow mentions, cited pages and citations on each AI platform with Visibility Overview in the AI Visibility Toolkit. Both toolkits sit in one subscription, so one trial covers them: try Semrush One free for 7 days.

The test

Our structured data study read the home pages of the top 1,000 websites on 10 November 2026. Our robots.txt study read their robots.txt files on 13 November 2026. On 14 November we joined them and kept the 437 sites where both succeeded: an HTML home page and a readable robots.txt.

A site counted as blocking an AI bot only if its robots.txt names that bot and disallows the whole site (14 bots, including GPTBot, ClaudeBot and PerplexityBot). A blanket block for all crawlers does not count. 94 of the 437 sites (21.5%) block at least one AI bot by name, and 64 (14.6%) block GPTBot.

Organization markup goes with slightly less blocking

Marked-up sites block AI bots a little less, or not at all. Share of 437 top sites whose robots.txt blocks at least one AI bot by name, for sites with and without each kind of markup: any JSON-LD 22.7% against 20.1%; Organization markup 18.9% against 23.1%; sameAs links 18.2% against 23.1%.
Marked-up sites block AI bots a little less, or not at all. Share of 437 top sites whose robots.txt blocks at least one AI bot by name, for sites with and without each kind of markup: any JSON-LD 22.7% against 20.1%; Organization markup 18.9% against 23.1%; sameAs links 18.2% against 23.1%.

Of the 169 sites with Organization markup, 32 (18.9%) block an AI bot by name. Of the 268 without it, 62 (23.1%) do: a gap of 4.2 points. For GPTBot specifically it is 11.2% against 16.8%. Sites whose markup includes sameAs links to other profiles (143) show 18.2% against 23.1% (294), and 11.2% against 16.3% for GPTBot.

Plain JSON-LD does not

The pattern breaks when the markup is any JSON-LD at all. Of the 233 sites with some JSON-LD, 22.7% block an AI bot by name. Of the 204 without any, 20.1% do. That runs the other way, by 2.6 points. For GPTBot alone it points back in the first direction (13.3% against 16.2%).

Three comparisons, two pointing one way and one the other, with gaps of three to five points on a few hundred sites each: that is not a relationship we would build a plan on.

What this cannot tell you

  • That markup changes blocking, or blocking changes markup. These are shares across sites. Larger or more media-focused companies may differ on both.
  • That the gaps are real. With 437 sites, differences of two to five points are within what chance can produce. The honest summary is "weak and inconsistent".
  • That blocking stops citation. We did not test whether any site is cited by an AI engine.
  • That the dates line up. Markup was read on 10 November and robots.txt on 13 November, and sites change.
  • That the whole web looks like this. These are top-ranked sites, read on their home pages only. Markup elsewhere on a site is not counted.

What to do with it

  1. Do not read markup as a statement about crawler access. A site can carry rich markup and still shut AI bots out.
  2. Check your own robots.txt for named AI bots before you assume your pages are readable to them.
  3. Decide per bot. Blocking a training crawler and allowing a search crawler are different choices.
  4. Keep the two audits separate: what your pages say about you, and who is allowed to read them.

Check the markup, check the access, track the result

Check your markup. Site Audit in the SEO Toolkit has a Markup report that counts the pages with structured data and flags invalid items.

Semrush's own example of Site Audit's Markup report, from its Knowledge Base (retrieved 29 October 2026): 881 pages checked, 99% carrying markup, 683 with valid structured data and none invalid, broken down by item type. Semrush's example site, not ours.
Semrush's own example of Site Audit's Markup report, from its Knowledge Base (retrieved 29 October 2026): 881 pages checked, 99% carrying markup, 683 with valid structured data and none invalid, broken down by item type. Semrush's example site, not ours.

Check what AI bots can reach. AI Search Health in Site Audit lists pages blocked from AI search bots.

Semrush's own illustration of the Site Audit overview, from its AI Search Health Knowledge Base page (retrieved 29 October 2026): a demo audit with JavaScript rendering disabled, Site Health 65%, crawled pages by status, an AI Search Health score of 58% and the pages blocked from AI search bots such as ChatGPT-User, OAI-SearchBot and Googlebot. Semrush's illustration, not our account.
Semrush's own illustration of the Site Audit overview, from its AI Search Health Knowledge Base page (retrieved 29 October 2026): a demo audit with JavaScript rendering disabled, Site Health 65%, crawled pages by status, an AI Search Health score of 58% and the pages blocked from AI search bots such as ChatGPT-User, OAI-SearchBot and Googlebot. Semrush's illustration, not our account.

See what AI says about you. Brand Performance shows how AI platforms describe a brand: sentiment, narratives, topics.

Semrush's own illustration of the Share of Voice widget in the AI Visibility Toolkit's Brand Performance reports, from its Knowledge Base (retrieved 13 October 2026): each brand's share of voice listed beside a line chart of the same shares over several weeks, with a "Losing visibility" note. Semrush's illustration, not our account.
Semrush's own illustration of the Share of Voice widget in the AI Visibility Toolkit's Brand Performance reports, from its Knowledge Base (retrieved 13 October 2026): each brand's share of voice listed beside a line chart of the same shares over several weeks, with a "Losing visibility" note. Semrush's illustration, not our account.

Track whether access turns into citations. Visibility Overview shows mentions, citations and cited pages over time.

Semrush's own illustration of the Visibility Overview report in the AI Visibility Toolkit, from its Knowledge Base (retrieved 9 October 2026): an AI Visibility score of 36/100, an "All AI platforms" selector, and mentions, citations and cited pages over six months. Semrush's illustration, not our account.
Semrush's own illustration of the Visibility Overview report in the AI Visibility Toolkit, from its Knowledge Base (retrieved 9 October 2026): an AI Visibility score of 36/100, an "All AI platforms" selector, and mentions, citations and cited pages over six months. Semrush's illustration, not our account.

The free trial is attached to Semrush One, since the AI Visibility Toolkit has no trial of its own and Semrush One includes it alongside the SEO Toolkit. To audit your markup and follow your citations before paying, try Semrush One free for 7 days. Semrush's product page words the offer as "Unlimited access to all Semrush One tools" (read on 16 October 2026). We have not used the trial, and each figure in this article is from our own saved crawls, none from a Semrush account.

Semrush One's own product page: "Try free for 7 days" button with "Unlimited access to all Semrush One tools" beneath it. Captured from Semrush's page on 26 September 2026; the same wording was live on 16 October 2026.
Semrush One's own product page: "Try free for 7 days" button with "Unlimited access to all Semrush One tools" beneath it. Captured from Semrush's page on 26 September 2026; the same wording was live on 16 October 2026.
Semrush's official Semrush One graphic, from the Semrush Creator Accelerator affiliate resources: "Semrush One" at the centre of ten capabilities, including AI + SEO Visibility, Site Health + AI Search Auditing, AI Brand Narrative and Share of Voice. Semrush's own image.
Semrush's official Semrush One graphic, from the Semrush Creator Accelerator affiliate resources: "Semrush One" at the centre of ten capabilities, including AI + SEO Visibility, Site Health + AI Search Auditing, AI Brand Narrative and Share of Voice. Semrush's own image.

Labelled is not the same as open

Describing yourself to machines and letting machines in are separate decisions, and the top sites do not tie them together. Check both for your own site.

Try Semrush One free for 7 days

Site Audit in the SEO Toolkit and Visibility Overview in the AI Visibility Toolkit, together in Semrush One. Free for 7 days for new users.

Try Semrush One free for 7 days

How we counted

  • Data: the saved home-page structured data crawl of 10 November 2026 and the saved robots.txt crawl of 13 November 2026, matched site by site on 14 November 2026, without fetching anything new.
  • Kept: 437 sites with an HTML home page and a readable robots.txt.
  • Blocking: a named AI bot with "Disallow: /" and no allowance for the whole site. Blanket rules for all crawlers were not counted.
  • Markup: any JSON-LD block on the home page; an Organization item; an Organization item with sameAs links.
  • Gap: the share blocking among sites with the markup minus the share among those without, in percentage points.
  • Left out: any link to AI citations, and pages other than the home page.
  • Reproduce it: the script and its saved output are in research/r2-e9-markup-vs-bots/.

Explore More AI Tools

Browse our curated directory of 855+ verified AI tools.

Browse Directory

Some links may be affiliate links. We may earn a small commission at no extra cost to you.