Skip to content

Google Says You Can Ignore llms.txt. 129 of the Top 1,000 Sites Publish One Anyway, 18% of Those We Could Reach, and 28 Also Publish a Full Version

Sohail Akhtar
Researched by Sohail AkhtarTheToolsVerse
November 15, 20267 min read
Who publishes llms.txt: 18.0% of the 717 top-1,000 sites that responded, 12.5% in ranks 1 to 100, 20.8% in ranks 101 to 500, 17.0% in ranks 501 to 1,000, 3.9% also llms-full.txt, against 37.7% of AI tool company sites.

Disclosure

This article is promotional content produced as part of the Semrush Creator Accelerator programme. It contains affiliate links to Semrush — if you buy through them we may earn a commission at no extra cost to you — and we may also receive a reward from Semrush for publishing it. All measurements in this article are our own. Semrush did not supply, review or approve the data, and the conclusions we draw are ours.

The advice industry has a long list of "GEO" tactics, and llms.txt is the best known: a Markdown file at the root of a site that summarises it for language models. On 14 November 2026 we read what Google says about it in its Search Central guide Google's Guide to Optimizing for Generative AI Features on Google Search: "For Google Search, you can ignore tactics like 'chunking' content, creating unnecessary AI text files (like llms.txt), or pursuing inauthentic mentions." The same guide adds that it is fine to create and maintain one if you want to.

Whether it helps other AI systems is a separate question. Here we asked a simpler one: who actually publishes the file?

Spend the time on foundations you can measure

Check the technical foundations that decide whether pages can be crawled, indexed and cited with Site Audit in the SEO Toolkit. The AI Visibility Toolkit is part of the same subscription: try Semrush One free for 7 days.

The test

On 14 November 2026 we requested /llms.txt and /llms-full.txt from each of the 1,000 most popular websites in the Tranco ranking (the 5 November 2026 download). 717 sites answered at least one request. We counted a file only if it came back with a success status, was not a web page, and was not simply the same text the site returns for any .txt address. We checked that last point by asking each site that appeared to have a file for a made-up .txt address; three returned the same text and were removed.

One in six publishes one

129 of the 717 sites that responded (18.0%) serve a real llms.txt. 28 (3.9%) also serve an llms-full.txt, and none has a full version without the short one. That is 12.9% of all 1,000 sites, since 283 did not respond.

Who publishes an llms.txt? Share of sites with a real llms.txt, November 2026: AI tool company sites (613, our earlier study) 37.7%; top 1,000 sites that responded (717) 18.0%; ranks 1 to 100 (72 responded) 12.5%; ranks 101 to 500 (274) 20.8%; ranks 501 to 1,000 (371) 17.0%; also publish llms-full.txt (717) 3.9%.
Who publishes an llms.txt? Share of sites with a real llms.txt, November 2026: AI tool company sites (613, our earlier study) 37.7%; top 1,000 sites that responded (717) 18.0%; ranks 1 to 100 (72 responded) 12.5%; ranks 101 to 500 (274) 20.8%; ranks 501 to 1,000 (371) 17.0%; also publish llms-full.txt (717) 3.9%.

The most popular sites are the least likely to have one: 9 of the 72 that responded in ranks 1 to 100 (12.5%), against 57 of 274 in ranks 101 to 500 (20.8%) and 63 of 371 in ranks 501 to 1,000 (17.0%).

AI tool companies publish it twice as often

In our earlier study of AI tool company sites, 231 of 613 (37.7%) published a real file. The sites that sell to people building with AI adopt the format at about twice the rate of the web's biggest sites.

What the files look like

120 of the 129 files (93.0%) start with a Markdown heading, as the format asks. The median file is 11,913 bytes, and the largest is 3,358,637 bytes. The first lines of the ones we saw are the same kind of text: a company name, a one-line description, then links.

117 sites, the three above included, returned a "success" status for /llms.txt but gave us a web page or the same text for any address. A request that succeeds is not evidence that a file exists.

What this cannot tell you

  • That the file helps or hurts. We counted publishers, not results. Google says it is unnecessary for Google Search. We did not test any other system.
  • That the sites we could not reach have none. 283 sites did not answer our requests.
  • How the files are used. In our earlier study, AI crawlers made thousands of requests to our own site and none for llms.txt, but that is one small site.
  • That the contents are good. We checked the shape, not the quality.
  • That the figures are fixed. This is one day, and the format is new.

What to do with it

  1. Do not buy the story that it is required. Most of the biggest sites do not have one, and Google's own guide says you can ignore it for Search.
  2. If you publish one, keep it honest and current. A stale summary is worse than none.
  3. Spend the time first on what Google says matters: crawlable, indexable pages with useful content.
  4. Test claims before repeating them. A tactic's popularity in the advice industry is not evidence of its effect.
  5. Measure your own AI visibility before and after any change.

Check the foundations, find the topics, measure visibility

Check technical foundations. Site Audit lists crawlability and indexability issues, with AI Search Health for pages blocked from AI bots.

Semrush's own illustration of the Site Audit overview, from its AI Search Health Knowledge Base page (retrieved 29 October 2026): a demo audit with JavaScript rendering disabled, Site Health 65%, crawled pages by status, an AI Search Health score of 58% and the pages blocked from AI search bots such as ChatGPT-User, OAI-SearchBot and Googlebot. Semrush's illustration, not our account.
Semrush's own illustration of the Site Audit overview, from its AI Search Health Knowledge Base page (retrieved 29 October 2026): a demo audit with JavaScript rendering disabled, Site Health 65%, crawled pages by status, an AI Search Health score of 58% and the pages blocked from AI search bots such as ChatGPT-User, OAI-SearchBot and Googlebot. Semrush's illustration, not our account.

Find the topics people search for. The Keyword Magic Tool in the SEO Toolkit groups related keywords and shows intent and volume.

Semrush's own illustration of the Keyword Magic Tool, from its Knowledge Base (retrieved 29 October 2026): keyword groups such as snowboard, boots and burton in the left sidebar beside a keyword list with intent, volume, KD and related percentage. Semrush's illustration, not our account.
Semrush's own illustration of the Keyword Magic Tool, from its Knowledge Base (retrieved 29 October 2026): keyword groups such as snowboard, boots and burton in the left sidebar beside a keyword list with intent, volume, KD and related percentage. Semrush's illustration, not our account.

See where rivals win visibility. Keyword Gap lists the keywords competitors rank for and you do not.

Semrush's own illustration of the Keyword Gap report, from its Knowledge Base (retrieved 29 October 2026): five root domains compared on organic keywords, with a Top Opportunities list of missing keywords and a Keyword Overlap diagram. Semrush's illustration, not our account.
Semrush's own illustration of the Keyword Gap report, from its Knowledge Base (retrieved 29 October 2026): five root domains compared on organic keywords, with a Top Opportunities list of missing keywords and a Keyword Overlap diagram. Semrush's illustration, not our account.

Measure your AI visibility. Visibility Overview in the AI Visibility Toolkit shows mentions, citations and cited pages across AI platforms.

Semrush's own illustration of the Visibility Overview report in the AI Visibility Toolkit, from its Knowledge Base (retrieved 9 October 2026): an AI Visibility score of 36/100, an "All AI platforms" selector, and mentions, citations and cited pages over six months. Semrush's illustration, not our account.
Semrush's own illustration of the Visibility Overview report in the AI Visibility Toolkit, from its Knowledge Base (retrieved 9 October 2026): an AI Visibility score of 36/100, an "All AI platforms" selector, and mentions, citations and cited pages over six months. Semrush's illustration, not our account.

The trial is Semrush One's; the AI Visibility Toolkit has no trial of its own and is included in Semrush One alongside the SEO Toolkit. To test your foundations and measure visibility without paying first, try Semrush One free for 7 days. Semrush's product page says "Unlimited access to all Semrush One tools" (read on 16 October 2026). That is an offer Semrush publishes and we have not tried, and no figure here came from a Semrush account.

Semrush One's own product page: "Try free for 7 days" button with "Unlimited access to all Semrush One tools" beneath it. Captured from Semrush's page on 26 September 2026; the same wording was live on 16 October 2026.
Semrush One's own product page: "Try free for 7 days" button with "Unlimited access to all Semrush One tools" beneath it. Captured from Semrush's page on 26 September 2026; the same wording was live on 16 October 2026.
Semrush's official Semrush One graphic, from the Semrush Creator Accelerator affiliate resources: "Semrush One" at the centre of ten capabilities, including AI + SEO Visibility, Site Health + AI Search Auditing, AI Brand Narrative and Share of Voice. Semrush's own image.
Semrush's official Semrush One graphic, from the Semrush Creator Accelerator affiliate resources: "Semrush One" at the centre of ten capabilities, including AI + SEO Visibility, Site Health + AI Search Auditing, AI Brand Narrative and Share of Voice. Semrush's own image.

A tactic that gets talked about a lot and is used by one site in six is still unproven. Measure first, copy second.

Try Semrush One free for 7 days

Site Audit, the Keyword Magic Tool and Keyword Gap in the SEO Toolkit, with Visibility Overview in the AI Visibility Toolkit, together in Semrush One. Free for 7 days for new users.

Try Semrush One free for 7 days

How we counted

  • Sites: Tranco ranks 1 to 1,000, from the list saved on 5 November 2026. Each site was asked for /llms.txt and /llms-full.txt once on 14 November 2026, with the www form as a fallback. A site "responded" if either request returned any HTTP status.
  • A real file: status 200, a content type that is not HTML, a body that does not start with an opening angle bracket, and a body that differs from what the site returns for a made-up .txt address (checked for every site that passed the first tests).
  • Heading check: the first line of the file is a Markdown heading that starts with "# ".
  • Quote: read from Google's English Search Central guide on 14 November 2026.
  • AI tool sites: the 37.7% figure (231 of 613) is from our earlier published study.
  • Left out: the quality of the files, and whether any crawler uses them.
  • Reproduce it: the crawler, the check and the analysis are in research/r2-e6-llms-txt-top1000/; the raw responses are not published.

Explore More AI Tools

Browse our curated directory of 855+ verified AI tools.

Browse Directory

Some links may be affiliate links. We may earn a small commission at no extra cost to you.