Top Sites With Organization Markup Block AI Crawlers Slightly Less, 18.9% Against 23.1%, but Plain JSON-LD Shows No Such Pattern: Joining Two of Our Studies

Disclosure
This article is promotional content produced as part of the Semrush Creator Accelerator programme. It contains affiliate links to Semrush — if you buy through them we may earn a commission at no extra cost to you — and we may also receive a reward from Semrush for publishing it. All measurements in this article are our own. Semrush did not supply, review or approve the data, and the conclusions we draw are ours.
Two pieces of advice often sit side by side. Describe your organisation in structured data so machines can understand who you are. And decide which AI crawlers may read your pages at all. They are usually discussed separately, yet they seem to pull in one direction: a site that labels itself for machines presumably wants machines to read it.
Does that show up in practice? We had both measurements for the same websites, so we joined them.
See whether AI platforms use your pages
Follow mentions, cited pages and citations on each AI platform with Visibility Overview in the AI Visibility Toolkit. Both toolkits sit in one subscription, so one trial covers them: try Semrush One free for 7 days.
The test
Our structured data study read the home pages of the top 1,000 websites on 10 November 2026. Our robots.txt study read their robots.txt files on 13 November 2026. On 14 November we joined them and kept the 437 sites where both succeeded: an HTML home page and a readable robots.txt.
A site counted as blocking an AI bot only if its robots.txt names that bot and disallows the whole site (14 bots, including GPTBot, ClaudeBot and PerplexityBot). A blanket block for all crawlers does not count. 94 of the 437 sites (21.5%) block at least one AI bot by name, and 64 (14.6%) block GPTBot.
Organization markup goes with slightly less blocking
Of the 169 sites with Organization markup, 32 (18.9%) block an AI bot by name. Of the 268 without it, 62 (23.1%) do: a gap of 4.2 points. For GPTBot specifically it is 11.2% against 16.8%. Sites whose markup includes sameAs links to other profiles (143) show 18.2% against 23.1% (294), and 11.2% against 16.3% for GPTBot.
Plain JSON-LD does not
The pattern breaks when the markup is any JSON-LD at all. Of the 233 sites with some JSON-LD, 22.7% block an AI bot by name. Of the 204 without any, 20.1% do. That runs the other way, by 2.6 points. For GPTBot alone it points back in the first direction (13.3% against 16.2%).
Three comparisons, two pointing one way and one the other, with gaps of three to five points on a few hundred sites each: that is not a relationship we would build a plan on.
What this cannot tell you
- That markup changes blocking, or blocking changes markup. These are shares across sites. Larger or more media-focused companies may differ on both.
- That the gaps are real. With 437 sites, differences of two to five points are within what chance can produce. The honest summary is "weak and inconsistent".
- That blocking stops citation. We did not test whether any site is cited by an AI engine.
- That the dates line up. Markup was read on 10 November and robots.txt on 13 November, and sites change.
- That the whole web looks like this. These are top-ranked sites, read on their home pages only. Markup elsewhere on a site is not counted.
What to do with it
- Do not read markup as a statement about crawler access. A site can carry rich markup and still shut AI bots out.
- Check your own robots.txt for named AI bots before you assume your pages are readable to them.
- Decide per bot. Blocking a training crawler and allowing a search crawler are different choices.
- Keep the two audits separate: what your pages say about you, and who is allowed to read them.
Check the markup, check the access, track the result
Check your markup. Site Audit in the SEO Toolkit has a Markup report that counts the pages with structured data and flags invalid items.

Check what AI bots can reach. AI Search Health in Site Audit lists pages blocked from AI search bots.

See what AI says about you. Brand Performance shows how AI platforms describe a brand: sentiment, narratives, topics.

Track whether access turns into citations. Visibility Overview shows mentions, citations and cited pages over time.

The free trial is attached to Semrush One, since the AI Visibility Toolkit has no trial of its own and Semrush One includes it alongside the SEO Toolkit. To audit your markup and follow your citations before paying, try Semrush One free for 7 days. Semrush's product page words the offer as "Unlimited access to all Semrush One tools" (read on 16 October 2026). We have not used the trial, and each figure in this article is from our own saved crawls, none from a Semrush account.


Labelled is not the same as open
Describing yourself to machines and letting machines in are separate decisions, and the top sites do not tie them together. Check both for your own site.
Try Semrush One free for 7 days
Site Audit in the SEO Toolkit and Visibility Overview in the AI Visibility Toolkit, together in Semrush One. Free for 7 days for new users.
Try Semrush One free for 7 daysHow we counted
- Data: the saved home-page structured data crawl of 10 November 2026 and the saved robots.txt crawl of 13 November 2026, matched site by site on 14 November 2026, without fetching anything new.
- Kept: 437 sites with an HTML home page and a readable robots.txt.
- Blocking: a named AI bot with "Disallow: /" and no allowance for the whole site. Blanket rules for all crawlers were not counted.
- Markup: any JSON-LD block on the home page; an Organization item; an Organization item with sameAs links.
- Gap: the share blocking among sites with the markup minus the share among those without, in percentage points.
- Left out: any link to AI citations, and pages other than the home page.
- Reproduce it: the script and its saved output are in
research/r2-e9-markup-vs-bots/.
Some links may be affiliate links. We may earn a small commission at no extra cost to you.
Related Articles
Explore more guides and reviews from our experts.

We Re-Checked 2,098 URLs That Five AI Engines Cited: 91.6% Still Load, 0.8% Are Gone, and 7.7% Cannot Be Verified by a Bot

No Domain Holds More Than 1.9% of AI Citations, 67% of Cited Domains Appear Once, and Wikipedia Was Never Cited: The Long Tail Behind AI Brand Consensus
