Skip to content

Google Says a Page Needs a Snippet to Appear in AI Features. On the Top 1,000 Sites, None of 551 Readable Home Pages Blocks One, 7 Are Marked noindex and 43 Hide Parts From Snippets

Sohail Akhtar
Researched by Sohail AkhtarTheToolsVerse
November 15, 20267 min read
Snippet directives on the top 1,000 sites' home pages: of 551 readable pages, none blocks snippets, 1.3% are noindex, 7.8% use data-nosnippet, 9.1% allow unlimited snippets and 15.4% ask for large image previews.

Disclosure

This article is promotional content produced as part of the Semrush Creator Accelerator programme. It contains affiliate links to Semrush — if you buy through them we may earn a commission at no extra cost to you — and we may also receive a reward from Semrush for publishing it. All measurements in this article are our own. Semrush did not supply, review or approve the data, and the conclusions we draw are ours.

Google's guide to generative AI features on Search is mostly a reminder that the basics still apply. One sentence in it is a hard rule, though. On 14 November 2026 we read, in Google's Search Central guide Google's Guide to Optimizing for Generative AI Features on Google Search: "To be eligible to be shown in generative AI features on Google Search, a page must be indexed and eligible to be shown in Google Search with a snippet".

Two things can take a page out of that rule: a directive that keeps it out of the index, and a directive that asks Google not to show a snippet. Both are one line in the page's head. So we asked how many of the biggest sites in the world use them.

Find pages that are noindexed or blocked

Crawl your site the way search engines and AI bots do, and see which pages are noindexed or blocked, with Site Audit in the SEO Toolkit. To get it with the AI Visibility Toolkit in one subscription, try Semrush One free for 7 days.

The test

We took the 1,000 most popular websites in the Tranco ranking (the list we downloaded on 5 November 2026, the same one we used in our structured data study) and requested each home page on 13 November 2026. 551 returned an HTML page. The others refused us, timed out, redirected to something that was not HTML or answered with an error, so every figure below is about those 551.

On each page we read the robots, googlebot and google meta tags, the X-Robots-Tag response header and any data-nosnippet attributes.

None of them blocks snippets

Snippet directives on the home pages of the top 1,000 sites that returned HTML (551), checked 13 November 2026: any robots directive 38.5% (212); max-image-preview large 15.4% (85); max-snippet -1, no limit, 9.1% (50); data-nosnippet on part of the page 7.8% (43); noindex 1.3% (7); nosnippet or max-snippet 0, 0% (0).
Snippet directives on the home pages of the top 1,000 sites that returned HTML (551), checked 13 November 2026: any robots directive 38.5% (212); max-image-preview large 15.4% (85); max-snippet -1, no limit, 9.1% (50); data-nosnippet on part of the page 7.8% (43); noindex 1.3% (7); nosnippet or max-snippet 0, 0% (0).

Not one of the 551 home pages used nosnippet or max-snippet:0. None set a snippet length limit either. The directive that Google's rule is about is, on these pages, simply not used.

What the pages did say about snippets:

  • 50 (9.1%) say max-snippet:-1, which allows a snippet of any length.
  • 43 (7.8%) contain a data-nosnippet attribute, which hides one part of the page from snippets and leaves the rest eligible. They include apple.com, linkedin.com and mozilla.org.
  • 85 (15.4%) ask for large image previews with max-image-preview:large.
  • 212 (38.5%) set some robots directive at all, including index, follow, noarchive or notranslate. 13 pages (2.4%) send an X-Robots-Tag header.

The real exclusion is noindex

7 of the 551 home pages (1.3%) carried noindex, 5 in a meta tag and 2 in an X-Robots-Tag header. A page that is not indexed cannot meet Google's rule, whatever its snippet settings. Some of these may be redirect or login pages that were never meant to be indexed, since we saw whatever the home address returned to us, so treat the figure as a lower bound on pages that are meant to be found.

Big sites are less likely to ask for large images

Large image previews are a different decision from snippets, but they show how much attention sites give the head of the page. Only 2 of the 62 readable sites in ranks 1 to 100 (3.2%) set max-image-preview:large, against 32 of 206 (15.5%) in ranks 101 to 500 and 51 of 283 (18.0%) in ranks 501 to 1,000. The sites with the most traffic are the least likely to set the directive.

What this cannot tell you

  • That a page shows up in Google's AI features. We checked the directives that could keep it out. We did not test whether any page actually appears there.
  • How the rest of each site behaves. We looked at home pages. A product, article or help page may carry different directives.
  • What the other 449 sites do. They did not return a readable HTML home page to our request, so nothing here describes them.
  • That the directives are the same for Google. A site can send different headers to different clients and countries. We fetched each page once, from one location, with an ordinary browser-style request that identified our research project.
  • Whether robots.txt blocks. We did not read robots.txt, which is a separate control.

What to do with it

  1. Check your own pages for noindex and nosnippet. The rule is one sentence, and a stray template directive can take a whole section out.
  2. Look at the headers too. Two of the seven noindex pages used an X-Robots-Tag header, which does not show in the page source.
  3. Use data-nosnippet on purpose. It hides a part of a page, such as a price or a paywalled paragraph, and leaves the rest of the page eligible.
  4. Do not copy the biggest sites. The sites with the most traffic here barely use preview directives, which says little about what a smaller site should do.
  5. Re-check after template changes. Directives travel with templates, and a release can add or remove them for thousands of pages at once.

Find the pages, check access, find the questions

Find pages that are noindexed, blocked or hard to crawl. Site Audit in the SEO Toolkit lists crawlability and indexability issues, including pages that AI search bots cannot reach.

Semrush's own illustration of Site Audit's issues list, from its Site Audit feature page (retrieved 29 October 2026): errors, warnings and notices filtered into categories including Crawlability and AI Search, with an "About the issue / How to fix" panel. Semrush's illustration, not our site.
Semrush's own illustration of Site Audit's issues list, from its Site Audit feature page (retrieved 29 October 2026): errors, warnings and notices filtered into categories including Crawlability and AI Search, with an "About the issue / How to fix" panel. Semrush's illustration, not our site.

See which AI crawlers a page blocks. The Crawled Pages report can be filtered by blocked AI search bots.

Semrush's own illustration of Site Audit's Crawled Pages report filtered by "Blocked AI search bots: ChatGPT-User", from its Knowledge Base (retrieved 29 October 2026): two pages marked Blocked, each listing the AI agents blocked from it, including ChatGPT-User, OAI-SearchBot and Googlebot. Semrush's illustration, not our account.
Semrush's own illustration of Site Audit's Crawled Pages report filtered by "Blocked AI search bots: ChatGPT-User", from its Knowledge Base (retrieved 29 October 2026): two pages marked Blocked, each listing the AI agents blocked from it, including ChatGPT-User, OAI-SearchBot and Googlebot. Semrush's illustration, not our account.

Find the topics worth covering. The Keyword Magic Tool in the SEO Toolkit groups related keywords and shows intent and volume, so the questions a page should answer can be sorted by demand.

Semrush's own illustration of the Keyword Magic Tool, from its Knowledge Base (retrieved 29 October 2026): keyword groups such as snowboard, boots and burton in the left sidebar beside a keyword list with intent, volume, KD and related percentage. Semrush's illustration, not our account.
Semrush's own illustration of the Keyword Magic Tool, from its Knowledge Base (retrieved 29 October 2026): keyword groups such as snowboard, boots and burton in the left sidebar beside a keyword list with intent, volume, KD and related percentage. Semrush's illustration, not our account.

See which of your pages AI platforms cite. Visibility Overview in the AI Visibility Toolkit shows mentions, citations and cited pages over time across AI platforms.

Semrush's own illustration of the Visibility Overview report in the AI Visibility Toolkit, from its Knowledge Base (retrieved 9 October 2026): an AI Visibility score of 36/100, an "All AI platforms" selector, and mentions, citations and cited pages over six months. Semrush's illustration, not our account.
Semrush's own illustration of the Visibility Overview report in the AI Visibility Toolkit, from its Knowledge Base (retrieved 9 October 2026): an AI Visibility score of 36/100, an "All AI platforms" selector, and mentions, citations and cited pages over six months. Semrush's illustration, not our account.

The free trial belongs to Semrush One, not to the AI Visibility Toolkit, which has none of its own; Semrush One includes that toolkit together with the SEO Toolkit. To audit your pages and see your citations without paying first, try Semrush One free for 7 days. Semrush's product page promises "Unlimited access to all Semrush One tools" (we read it on 16 October 2026). That is Semrush's published offer, which we did not test, and none of the data in this article came from a Semrush account.

Semrush One's own product page: "Try free for 7 days" button with "Unlimited access to all Semrush One tools" beneath it. Captured from Semrush's page on 26 September 2026; the same wording was live on 16 October 2026.
Semrush One's own product page: "Try free for 7 days" button with "Unlimited access to all Semrush One tools" beneath it. Captured from Semrush's page on 26 September 2026; the same wording was live on 16 October 2026.
Semrush's official Semrush One graphic, from the Semrush Creator Accelerator affiliate resources: "Semrush One" at the centre of ten capabilities, including AI + SEO Visibility, Site Health + AI Search Auditing, AI Brand Narrative and Share of Voice. Semrush's own image.
Semrush's official Semrush One graphic, from the Semrush Creator Accelerator affiliate resources: "Semrush One" at the centre of ten capabilities, including AI + SEO Visibility, Site Health + AI Search Auditing, AI Brand Narrative and Share of Voice. Semrush's own image.

A one-line rule

Google's rule for its AI features is short, and the biggest sites already meet it. Check that you do too, on every template.

Try Semrush One free for 7 days

Site Audit and the Keyword Magic Tool in the SEO Toolkit, with Visibility Overview in the AI Visibility Toolkit, together in Semrush One. Free for 7 days for new users.

Try Semrush One free for 7 days

How we counted

  • Population: Tranco ranks 1 to 1,000 (downloaded 5 November 2026). One request to each domain's home page on 13 November 2026, falling back to the www address when needed. 551 returned an HTML page with a 200 status.
  • Directives: the content of every meta tag named robots, googlebot or google (the news-only googlebot-news tag was ignored), plus the X-Robots-Tag header. "Blocks snippets" means nosnippet or max-snippet:0. "No limit" means max-snippet:-1. "Noindex" means noindex or none.
  • Parts of a page: a page counts for data-nosnippet if the attribute appears anywhere in its HTML.
  • Rank bands: ranks 1 to 100 (62 readable pages), 101 to 500 (206) and 501 to 1,000 (283).
  • The quote: read from Google's English Search Central guide on 14 November 2026.
  • Not shown: robots.txt, other pages of each site, and whether any page appears in a Google AI feature.
  • Reproduce it: the crawler and the analysis are in our research folder (research/r2-d8-snippet-controls/); the raw crawl is not published.

Explore More AI Tools

Browse our curated directory of 855+ verified AI tools.

Browse Directory

Some links may be affiliate links. We may earn a small commission at no extra cost to you.