38% of AI Tool Companies Publish llms.txt. In a Week of Our Logs, ChatGPT and Claude Never Opened Ours

Google's own guide, Optimizing your website for generative AI features on Google Search (last updated 10 July 2026), contains a sentence that settles one question and leaves another open: "It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

So the file does nothing for Google. The open question is the one in the middle of that sentence — which "other services or systems" actually use it? We run an AI tools directory, which puts us in a good position to answer it from both ends: we can see what the companies we list are writing, and we can see, in our own logs, who comes to read the llms.txt we publish ourselves.

The writing side: 613 AI tool companies

Our catalogue links to the websites of 884 AI tools. Leaving out tools hosted on GitHub, Hugging Face and app stores, that is 804 distinct company websites. On 28 October 2026 we requested the home page and /llms.txt of each; 613 home pages answered.

We were strict about what counts. A real llms.txt has to come back with HTTP 200, must not be a web page, and must start with a Markdown heading, the way the llms.txt proposal lays it out.

231 of the 613 — 37.7% — publish a real llms.txt. Among them on 28 October: Descript, ElevenLabs, HeyGen, HubSpot, Jasper, Notion, Synthesia, Zapier and Semrush itself. Among the companies without one at the root of their main domain: Canva, Grammarly, Ahrefs, Anthropic and OpenAI, which all return not found — and Otter.ai, which answers with a web page.

The number we did not expect is the second bar. 95 sites (15.5%) answer /llms.txt with an ordinary web page and a 200 status — usually their home page or a styled "not found" page that does not say so. To a script that only checks the status code, every one of those looks like a site with an llms.txt. Any audit or adoption figure that counts status codes alone will be inflated by about a third.

What is inside the 231 files

These are not token gestures. The real files we collected are substantial and, mostly, follow the proposal:

Most AI tool companies use the file the way the format intends — a short summary, then a map of their docs, pricing and product pages. That makes the second half of this study more interesting, not less: a lot of careful work is going into these files.

The reading side: one week of our own logs

TheToolsVerse publishes an llms.txt and an llms-full.txt, generated from the same catalogue as the site. So we asked Cloudflare which user agents requested them between 21 and 28 October 2026, and set that against how often the same agents requested anything else on the site.

OpenAI's three crawlers — GPTBot, OAI-SearchBot and ChatGPT-User — made 3,022 requests to our site that week. Not one was for llms.txt. Anthropic's three made 2,473. Not one was for llms.txt either. Perplexity's crawlers made 4,017 requests and fetched llms-full.txt once. Googlebot requested each file once in 2,594 visits — the way it fetches any URL it finds linked, which is consistent with Google's statement that Search ignores the content. Bing's crawler, whose index several AI assistants draw on, made 10,967 requests and none for either file.

It is not that these crawlers skip housekeeping files. In the same week ClaudeBot fetched our robots.txt 155 times and OAI-SearchBot 77 times. They read the file that tells them what they may crawl. They did not read the one that tries to tell them what matters.

So who did read it?

Our two files were requested 97 times in the week (67 for llms.txt, 30 for llms-full.txt). The largest single reader was ExaSearchBot, the crawler of Exa, a search API built for AI applications (11 requests). After it came a long tail: SEO and site-audit crawlers (SEOJuice, Semrush's SiteAuditBot, Ahrefs Site Audit, BuiltWith), new "agentic web" registries that map sites for AI agents (Andi's agent probe, agentwebmap, BrickBlue's agentic-web registry, LinkupBot), a few brand-monitoring tools, and a few dozen requests carrying ordinary browser user agents — people looking at the file, or tools that do not identify themselves.

That is the honest shape of the audience for llms.txt today, at least on one AI-tools website: developer-facing retrieval APIs, agent registries and the tools that audit websites — not the assistants most people mean when they say "AI search".

What this means if you are writing one

• It costs little, so keep it if you have it. Google's guide says it neither helps nor harms in Search, and some retrieval systems do read it. Our own file is generated from data we keep anyway.
• Do not count it as AI visibility work. In a week of logs, ChatGPT's and Claude's crawlers fetched thousands of our pages and none of our llms.txt. The pages are what they read — so the pages are what has to be good.
• Check it the way a crawler would. If /llms.txt returns your home page with a 200, you have told every audit tool you have a file you do not have.
• Measure the outcome. The question that matters is not whether the file exists but whether AI answers mention and cite you. An earlier study of ours found that Google AI Mode cited pages on sites with an llms.txt no more often than other pages; the logs above suggest why.

Measure what AI search does with your site

Google's guide is blunt about the alternative to tactics like this: "For Google Search, you can ignore tactics like "chunking" content, creating unnecessary AI text files (like llms.txt), or pursuing inauthentic mentions." What is left is measurement — of whether you are cited, and of whether crawlers can reach the pages that would earn it.

Check what crawlers can actually reach. Site Audit reports an AI Search Health score — Semrush's Knowledge Base describes it as "a score that reflects how optimized your website is for AI search" — and lists the AI crawlers your robots.txt blocks. Its technical-readiness check includes llms.txt among other elements; read that line as housekeeping, and the blocked-crawler list as the part that decides whether AI systems see your pages at all.

See whether you are cited, prompt by prompt. Visibility Overview, in the AI Visibility Toolkit, shows the prompts where AI answers mention or cite your brand, with a preview of each answer and the sources behind it — the measurement our log study cannot give you.

Track AI answers next to rankings. Position Tracking follows your keywords daily, including whether they trigger an AI Overview, and with Semrush One or the AI Visibility Toolkit it follows your visibility for chosen prompts on AI platforms as well.

Find where competitors show up and you do not. Keyword Gap compares your domain's keywords with up to four competitors' and lists the ones you are missing — content the crawlers in our logs would actually come and read.

Benchmark the pages that win instead. Organic Rankings (the report formerly called Organic Research) shows which keywords and pages bring a competitor its organic traffic and how that changes month to month. That shows which pages are earning visibility, which no text file can tell you.

Semrush One bundles the SEO Toolkit and the AI Visibility Toolkit, and the 7-day free trial is Semrush One's — the AI Visibility Toolkit has no free trial of its own. To run these checks on your own site before paying, try Semrush One free for 7 days: its page offers "Unlimited access to all Semrush One tools" and notes "Offer available to new users only", checked on 28 October 2026. Those are Semrush's published terms, not something we tested; none of the data above came from a Semrush account.

The file is written for readers who mostly aren't there yet

More than a third of the AI tool companies we list have written an llms.txt, most of them carefully. In our logs, the readers were retrieval APIs, agent registries and audit tools; the crawlers behind ChatGPT and Claude read thousands of our pages and none of our llms.txt. Keep the file if you have one. Spend the effort on the pages.

Method and sources

• Websites (804, 613 reachable): the distinct websites behind the 884 tools in the TheToolsVerse catalogue, excluding tools hosted on GitHub, Hugging Face and app stores. Tracking subdomains such as try. and get. were reduced to the company domain. "Reachable" means the home page answered with a status below 400.
• The check: one request each to https://<domain>/ and https://<domain>/llms.txt on 28 October 2026, following redirects, and /llms-full.txt where an llms.txt existed. A file counts as real only if it returned 200, was not HTML and began with a Markdown heading. Root domain only: a company whose file lives on a docs subdomain is counted as not having one.
• Logs: Cloudflare's analytics for thetoolsverse.com, 21 to 28 October 2026, matched on user-agent strings. Requests are not verified as genuine for every agent, and one week on one website is a narrow window — larger or differently-placed sites may see different readers.
• What this cannot show: whether an AI system uses an llms.txt it obtains some other way (for example through a developer API or a user pasting it in), or what any crawler does after fetching a file.

Written by Sohail Akhtar. The saved responses, log export and analysis script can be re-run.