Most AI Crawlers Don't Run JavaScript. We Rendered 515 AI Tool Home Pages Both Ways: 67 Show a Crawler Less Than Half Their Words

Google renders JavaScript before it indexes a page. Most AI crawlers do not. When Vercel analysed AI crawler traffic across its network, it reported that "none of the major AI crawlers currently render JavaScript", and that while ChatGPT's and Claude's crawlers "do fetch JavaScript files", "they can't read client-side rendered content" (Vercel, 17 December 2024).

That makes a simple measurement worth doing: how much of a page exists before any script runs? Whatever is missing from the raw HTML is missing from what those crawlers read. We measured it on the home pages of the AI tool companies we list — companies building some of the most JavaScript-heavy products on the web.

Every page, loaded twice

On 29 October 2026 we fetched each AI tool company home page in our catalogue twice:

1. Raw — a plain request, the way a non-rendering crawler receives it.
2. Rendered — in headless Chrome, with up to eight seconds for scripts to run, the way a person (or Googlebot) sees it.

For each page we counted the visible words both ways, compared which distinct words appeared, and checked where the <h1> headline turned up. 515 pages rendered with at least 100 words, enough to compare.

Most of these pages are fine: 350 of 515 (68%) already carry 90% or more of their text in the HTML, and the median page carries 98%. Server-side rendering and static generation are now common, and it shows.

But 67 pages (13%) show a non-rendering crawler less than half of their words — and 41 of those show less than a tenth. Those 41 are, to a crawler like GPTBot or ClaudeBot, essentially a title and an empty page: a navigation bar, a cookie notice, and nothing about what the product does.

Another 98 pages (19%) sit in between, typically with the core message in the HTML and tabs, carousels, pricing tables or testimonials added by scripts.

The headline that isn't there

The single most important line on a home page is usually the <h1>. On 30 of the pages, that headline exists only after JavaScript runs. A crawler reading the raw HTML gets no main heading at all — it has to guess what the page is about from whatever else survived.

What to check on your own pages

1. Compare raw and rendered. Open "View source" (not the element inspector) and search for your headline, your product description and your pricing. If they are not there, non-rendering crawlers cannot see them.
2. Server-render what matters. The headline, the one-paragraph description of the product, the main features and the prices belong in the initial HTML. Interactivity can load later.
3. Don't put the explanation in a carousel. Content that only exists in a script-driven slider or tab is often absent from the HTML.
4. Check robots.txt too. A page that renders perfectly is still invisible to an AI crawler you block.
5. Measure the effect. After a fix, watch whether AI answers start citing the page.

Audit the HTML, then watch the citations

See the site both ways. Site Audit can crawl with JavaScript rendering enabled or disabled, which turns the comparison above into a site-wide report, and its AI Search checks list the pages blocked from AI search bots such as ChatGPT-User and OAI-SearchBot.

Measure AI visibility by platform. Prompt Tracking, part of the AI Visibility Toolkit, follows one domain's visibility across ChatGPT, Google AI Mode, Gemini, Bing and Google side by side — the before-and-after view for a rendering fix.

See which pages AI cites — and why yours isn't. Prompt Tracking shows the cited pages for each prompt with details on each one, so you can check whether the pages AI prefers are the ones that render without scripts.

Keep traditional rankings in view. Position Tracking follows your keywords daily, and its SERP features filter tracks whether each keyword shows an AI Overview and whether your page is cited in it. That lets you measure a rendering fix in search and in AI answers at the same time.

Compare with the sites that do get read. Organic Rankings, formerly Organic Research, shows which pages bring a competitor its search traffic. If their winning pages are served as HTML and yours need JavaScript, you have found the gap.

Semrush One bundles the SEO Toolkit and the AI Visibility Toolkit, and the 7-day free trial is Semrush One's — the AI Visibility Toolkit has no free trial of its own. To audit your own pages before paying, try Semrush One free for 7 days: its page offers "Unlimited access to all Semrush One tools" and notes "Offer available to new users only", checked on 28 October 2026. Those are Semrush's published terms, not something we tested; none of the data above came from a Semrush account.

Two pages per URL

Every URL is really two pages: the one people see and the one a non-rendering crawler receives. For two thirds of AI tool home pages they are the same. For one in eight, the crawler's version is mostly empty — and that is the version the major AI crawlers read. View source is the cheapest AI-readability audit there is.

How the two loads were compared

• Pages (613 checked, 515 compared): the home page of each AI tool company in the TheToolsVerse catalogue whose website answered on 28 October 2026. Pages were compared when the raw request returned 200 and the rendered page had at least 100 words.
• Raw: one HTTP request with a desktop Chrome user agent, no JavaScript. Rendered: headless Chrome (--dump-dom, 8-second virtual time budget), same user agent, 29 October 2026.
• Words: visible text after removing scripts, styles, noscript and template blocks; a word = a token containing at least two letters. The share is raw words ÷ rendered words, capped at 100%. The headline check compares the first <h1> in each version.
• What a home-page test misses: how any specific AI crawler behaves today — Vercel's finding is from December 2024 and crawlers change — or whether missing text matters for a given answer. Some pages show different content to different visitors, and eight seconds may not be enough for every script.

Written by Sohail Akhtar. The saved pages and comparison script can be re-run.