Google Says You Can Ignore llms.txt. 129 of the Top 1,000 Sites Publish One Anyway, 18% of Those We Could Reach, and 28 Also Publish a Full Version

Disclosure
This article is promotional content produced as part of the Semrush Creator Accelerator programme. It contains affiliate links to Semrush — if you buy through them we may earn a commission at no extra cost to you — and we may also receive a reward from Semrush for publishing it. All measurements in this article are our own. Semrush did not supply, review or approve the data, and the conclusions we draw are ours.
The advice industry has a long list of "GEO" tactics, and llms.txt is the best known: a Markdown file at the root of a site that summarises it for language models. On 14 November 2026 we read what Google says about it in its Search Central guide Google's Guide to Optimizing for Generative AI Features on Google Search: "For Google Search, you can ignore tactics like 'chunking' content, creating unnecessary AI text files (like llms.txt), or pursuing inauthentic mentions." The same guide adds that it is fine to create and maintain one if you want to.
Whether it helps other AI systems is a separate question. Here we asked a simpler one: who actually publishes the file?
Spend the time on foundations you can measure
Check the technical foundations that decide whether pages can be crawled, indexed and cited with Site Audit in the SEO Toolkit. The AI Visibility Toolkit is part of the same subscription: try Semrush One free for 7 days.
The test
On 14 November 2026 we requested /llms.txt and /llms-full.txt from each of the 1,000 most popular websites in the Tranco ranking (the 5 November 2026 download). 717 sites answered at least one request. We counted a file only if it came back with a success status, was not a web page, and was not simply the same text the site returns for any .txt address. We checked that last point by asking each site that appeared to have a file for a made-up .txt address; three returned the same text and were removed.
One in six publishes one
129 of the 717 sites that responded (18.0%) serve a real llms.txt. 28 (3.9%) also serve an llms-full.txt, and none has a full version without the short one. That is 12.9% of all 1,000 sites, since 283 did not respond.
The most popular sites are the least likely to have one: 9 of the 72 that responded in ranks 1 to 100 (12.5%), against 57 of 274 in ranks 101 to 500 (20.8%) and 63 of 371 in ranks 501 to 1,000 (17.0%).
AI tool companies publish it twice as often
In our earlier study of AI tool company sites, 231 of 613 (37.7%) published a real file. The sites that sell to people building with AI adopt the format at about twice the rate of the web's biggest sites.
What the files look like
120 of the 129 files (93.0%) start with a Markdown heading, as the format asks. The median file is 11,913 bytes, and the largest is 3,358,637 bytes. The first lines of the ones we saw are the same kind of text: a company name, a one-line description, then links.
117 sites, the three above included, returned a "success" status for /llms.txt but gave us a web page or the same text for any address. A request that succeeds is not evidence that a file exists.
What this cannot tell you
- That the file helps or hurts. We counted publishers, not results. Google says it is unnecessary for Google Search. We did not test any other system.
- That the sites we could not reach have none. 283 sites did not answer our requests.
- How the files are used. In our earlier study, AI crawlers made thousands of requests to our own site and none for
llms.txt, but that is one small site. - That the contents are good. We checked the shape, not the quality.
- That the figures are fixed. This is one day, and the format is new.
What to do with it
- Do not buy the story that it is required. Most of the biggest sites do not have one, and Google's own guide says you can ignore it for Search.
- If you publish one, keep it honest and current. A stale summary is worse than none.
- Spend the time first on what Google says matters: crawlable, indexable pages with useful content.
- Test claims before repeating them. A tactic's popularity in the advice industry is not evidence of its effect.
- Measure your own AI visibility before and after any change.
Check the foundations, find the topics, measure visibility
Check technical foundations. Site Audit lists crawlability and indexability issues, with AI Search Health for pages blocked from AI bots.

Find the topics people search for. The Keyword Magic Tool in the SEO Toolkit groups related keywords and shows intent and volume.

See where rivals win visibility. Keyword Gap lists the keywords competitors rank for and you do not.

Measure your AI visibility. Visibility Overview in the AI Visibility Toolkit shows mentions, citations and cited pages across AI platforms.

The trial is Semrush One's; the AI Visibility Toolkit has no trial of its own and is included in Semrush One alongside the SEO Toolkit. To test your foundations and measure visibility without paying first, try Semrush One free for 7 days. Semrush's product page says "Unlimited access to all Semrush One tools" (read on 16 October 2026). That is an offer Semrush publishes and we have not tried, and no figure here came from a Semrush account.


Popular is not proven
A tactic that gets talked about a lot and is used by one site in six is still unproven. Measure first, copy second.
Try Semrush One free for 7 days
Site Audit, the Keyword Magic Tool and Keyword Gap in the SEO Toolkit, with Visibility Overview in the AI Visibility Toolkit, together in Semrush One. Free for 7 days for new users.
Try Semrush One free for 7 daysHow we counted
- Sites: Tranco ranks 1 to 1,000, from the list saved on 5 November 2026. Each site was asked for
/llms.txtand/llms-full.txtonce on 14 November 2026, with thewwwform as a fallback. A site "responded" if either request returned any HTTP status. - A real file: status 200, a content type that is not HTML, a body that does not start with an opening angle bracket, and a body that differs from what the site returns for a made-up
.txtaddress (checked for every site that passed the first tests). - Heading check: the first line of the file is a Markdown heading that starts with "# ".
- Quote: read from Google's English Search Central guide on 14 November 2026.
- AI tool sites: the 37.7% figure (231 of 613) is from our earlier published study.
- Left out: the quality of the files, and whether any crawler uses them.
- Reproduce it: the crawler, the check and the analysis are in
research/r2-e6-llms-txt-top1000/; the raw responses are not published.
Some links may be affiliate links. We may earn a small commission at no extra cost to you.
Related Articles
Explore more guides and reviews from our experts.

Top Sites With Organization Markup Block AI Crawlers Slightly Less, 18.9% Against 23.1%, but Plain JSON-LD Shows No Such Pattern: Joining Two of Our Studies

We Re-Checked 2,098 URLs That Five AI Engines Cited: 91.6% Still Load, 0.8% Are Gone, and 7.7% Cannot Be Verified by a Bot
