We Asked ChatGPT the Same 40 Questions Three Times. The Brands Held. The Sources Didn't.

In September Glen Allsopp published a 28-day study of AI answers on Detailed: more than 70,000 responses from ChatGPT and Google's AI Overviews, the same 1,300-plus prompts asked every day. His most quoted finding is about sources: "Only about a quarter of the pages ChatGPT cited for a prompt were cited again the next day." His most useful one is about brands: when the brand that usually leads a ChatGPT answer was not named first, "it usually hadn't slipped down the list, but was actually missing entirely."

A day-to-day comparison leaves one question open. If a page drops out of tomorrow's answer, did something change — or would it have dropped out if you had asked again a minute later? Those need opposite responses. One is a signal. The other is noise you should average away.

So we separated them. On 13 October 2026 we asked ChatGPT the same 40 questions we had asked it on 9 October, and then asked all 40 again straight away. That gives two comparisons: minutes apart, and four days apart.

The brand named first stayed the same on 28 of 30 questions across all three runs. The cited pages did not: run twice within minutes, only 54.2% of the pages ChatGPT cited came back the second time. Four days apart, 50.3% did.

Most of the change between 9 and 13 October was already there between two runs a few minutes apart. On this panel, most of what looks like citation volatility is the variation of a single answer, not drift.

How we measured

• Questions: the same 40 we used to test whether ChatGPT's citations come from Bing's top 20 and how far three AI engines agree: 15 "best/which" lists, 10 comparisons, 10 questions about one product, 5 how-tos, all about AI software, written before any answers were seen.
• Runs: ChatGPT with web search forced on, US English, through DataForSEO's LLM scraper. The model reported itself as gpt-6 in all three runs. Run 0: 9 October 2026 from 18:32 UTC. Run A: 13 October, 12:56–13:02 UTC. Run B: 13 October, 13:02–13:09 UTC, straight after A.
• Pages: the sources ChatGPT listed. A page counts as repeated if the same URL — protocol, www, query string and trailing slash ignored — was cited again for the same question.
• Brands: the brands ChatGPT's answer named, in the order it named them, as returned by the scraper. We merged spellings of the same product ("Fathom AI" and "Fathom", "Otter.ai" and "Otter"). 30 of the 40 questions named brands in all three runs. The other ten did not name brands in at least one run — seven of them were questions about a single product, such as "how much does midjourney cost per month?", where the answer is about that product rather than a choice between several — and they are left out of the brand figures.
• Validity: every response was checked for a successful status before it was used. All 120 ChatGPT responses were valid.

What we found

The brands are the stable part

On 28 of 30 questions the same brand came first in all three runs. When an answer named five tools, it usually named most of the same five next time, and rarely in a different order at the top.

The two exceptions show what the movement looks like when it happens. For "best ai coding assistant for vs code", ChatGPT put GitHub Copilot first on 9 October, Cursor first in run A, and GitHub Copilot first again in run B. For "how to make a video from text with ai", it went from Runway to Google Gemini and back to Runway. Neither was a brand losing its place; both were a close race answered differently on different draws.

This is steadier than Allsopp's broader panel, where the usual top brand was named first on about half of ChatGPT's days. Our questions are about one category, AI software, where most questions have an obvious market leader. A category with no clear leader will look more like his numbers than ours.

The sources are the unstable part

ChatGPT cited about eight pages per answer in every run — 7.8 on 9 October, 8.1 on both 13 October runs. The count barely moved. What changed was which eight.

Across the three runs, the 40 questions produced 591 distinct question-and-page pairs. Only 120 — 20.3% — were cited in all three runs. 345 — 58.4% — were cited once and never again. At website level the core is a little larger: 26.9% of cited websites appeared in all three runs, and half appeared in only one.

That is Allsopp's core-and-tail pattern, seen from a different angle. A small set of pages comes back every time. A long tail is sampled fresh with each answer.

Our pages repeated more often than his: about half came back rather than about a quarter. Our 40 questions sit in one category and ran on one model version, while his panel spans many categories over four weeks, so the gap is not surprising. The shape is the same.

Narrow questions are steadier than open ones

A question with one obvious source — the vendor's own pricing or help page — gets that source most of the time. Open questions have dozens of plausible sources, and ChatGPT picks a different handful each time. The "best" lists, where independent publishers compete, churned more than comparisons or single-product questions. That is also where a single screenshot is least likely to show what a buyer sees.

The search behind the answer churns even more

Each answer came with the pages ChatGPT retrieved while searching, not only the ones it cited — about 14 per answer. The retrieved sets overlapped by about 31% between runs, whether the runs were minutes or days apart.

The searches themselves rarely repeated. ChatGPT ran one or two searches per question, and the exact same search string came back on only 3 of 40 questions between the two 13 October runs. Across the three runs the 40 questions produced 198 different search strings.

Our own site is a small example of the whole picture. Our Photomath vs Symbolab comparison was ChatGPT's first source for a maths-solver question on 29 September. In this panel it was retrieved for the Photomath vs Symbolab question on 9 October and in run B — and cited in neither, nor in run A. A single check could have reported us as cited, retrieved-but-not-cited, or absent, depending on which answer it caught.

Gemini and AI Mode, four days apart

We repeated the same 40 questions on Gemini and Google AI Mode on 13 October and compared them with their 9 October answers.

Google AI Mode was the steadiest of the three: 63.7% of the pages it cited on 9 October were cited again on 13 October. Gemini was the least steady, at 41.5%, with an average page overlap of 27.8% per question. ChatGPT sat between them. We asked Gemini and AI Mode once on 13 October, not twice, so for those two engines the figures mix run-to-run noise with drift and cannot be split the way ChatGPT's can.

One data note matters here. On the first 13 October pass, 19 of AI Mode's 40 answers came back short and without any sources — a median answer on 9 October ran to about 3,000 characters, these to a few hundred. We treated them as incomplete captures, not as answers that cited nothing, and asked again. After two more passes, five still had no sources. Five of the 9 October answers had none either. For both Gemini and AI Mode, we left out every question with an empty answer on either date rather than count it as "everything changed".

What this means for reporting AI visibility

• Report the brand and the citation separately. On this panel, whether ChatGPT recommends you was stable across days. Whether it cites your page was not stable between two runs a few minutes apart. Merging them into one number buries a reliable signal under a noisy one.
• Never conclude anything from one answer. A page that disappears from one answer has roughly even odds of being back in the next. Run a prompt three times before you call a change.
• Prefer first-mention rate and presence over time to "position". Being named first held on 28 of 30 questions. That is a number worth tracking. A citation rank inside one answer is not.
• Expect more noise exactly where the opportunity is. Open "best" questions churned more than narrow ones, and they are the ones independent sites can win. Measure them with more runs, not fewer.
• Watch what is retrieved, not only what is cited. Being in the candidate set is the step before a citation, and our own page shows how often you can be there without being credited.

What we could not test

• Many days in a row. Two dates and two same-day runs separate noise from drift, but they cannot show a trend. Allsopp's 28-day design is the right one for that.
• Logged-in users and memory. Every answer was logged out, in US English. A signed-in user with history may see something else.
• Model changes. All three ChatGPT runs reported the same model. A model release would be a different experiment.
• Scale. 40 questions in one category is a case study. Its value is the same-hour control, which most volatility comparisons do not have.

Run this yourself

1. Fix a prompt set before you look at any answers — 20 to 40 questions your buyers ask, in their words.
2. Ask each prompt at least three times on day one, minutes apart. The spread between those runs is your noise floor.
3. Record brands and citations separately: who is named, in what order, and which pages are cited.
4. Repeat on a schedule, and only call a change when it is bigger than the day-one spread.
5. Track first-mention rate and presence across runs rather than any single answer's order.

Where Semrush fits

Step four is where doing this by hand stops scaling: our panel took 120 ChatGPT answers for one model, on two dates. Prompt Tracking, part of the AI Visibility Toolkit, is built for the scheduled version. Its Knowledge Base page, checked on 13 October 2026, says it "lets you monitor your brand's daily visibility for a custom set of prompts across AI search platforms" and names ChatGPT, Google AI Mode and Gemini. Daily records of a fixed prompt set are what turn the noise in our tables into a trend you can read.

Step five — presence over time instead of a single answer — is what the Brand Performance reports chart. Semrush's Knowledge Base describes the Share of Voice widget as one that "Compares your brand's overall visibility in AI answers against competitors, visualizing the market share of mentions for each brand", and notes that "The data in all Brand Performance reports is updated weekly." The same suite includes a Questions report, which "helps you gain insight into your audience's intent by analyzing the questions they ask AI."

For the unstable half — which pages get cited — Visibility Overview has a Cited Sources tab, defined in Semrush's words as "The websites that AI systems reference when generating answers." Read against our tables, that list is best treated as a frequency over many answers, not a fixed set: a website cited in most runs is in the core, one cited once is in the tail.

Step one, choosing the prompts, is the part most teams improvise. Prompt Research "helps you discover and prioritize the topics and prompts people use in AI-generated search", according to its Knowledge Base, and its Source Domains metric shows "The websites cited by AI platforms as sources in their responses, including the top three sources." We wrote our 40 questions from experience; a prompt set built from observed demand would weight the questions buyers actually ask.

Every tool above sits in Semrush One, which Semrush's own affiliate graphic sums up as a set of linked capabilities — Daily Tracking for Keywords & AI Prompts and Share of Voice among them.

A planning note: the free trial belongs to Semrush One, not to the AI Visibility Toolkit bought on its own. Semrush One's page offers "Try free for 7 days" with "Unlimited access to all Semrush One tools" — "Offer available to new users only", checked on 13 October 2026. Read that as Semrush's terms, not our test; every measurement above was made without a Semrush account.

What we actually learned

Asked the same 40 questions three times, ChatGPT named the same brand first on 28 of 30 questions — and cited a different mix of pages each time. Two runs a few minutes apart shared about half their cited pages. Runs four days apart were barely less alike.

That splits AI visibility into two different measurements. Whether ChatGPT recommends you is a stable signal you can track week to week. Whether it cites your page is a probability, and on this panel most of its movement was there before any time had passed. Treat the first as a KPI and the second as a rate across many answers, and stop letting a single screenshot decide either.

Sources and method

Written by Sohail Akhtar. Every figure here is a measurement on a stated date; the saved answers from every run and the comparison script can be re-run.