AI answer variability

AI answer variability is the tendency of AI assistants such as ChatGPT, Gemini, Microsoft Copilot or Google AI Mode to return a different answer each time the same prompt is asked: other brands, another order, a different number of recommendations and different cited sources.

Why one AI answer can't be trusted on its own

Checking a prompt once and keeping a screenshot tells you what one run said, not what the engine usually says. The gaps are wide. In SparkToro and Gumshoe's study of AI recommendations, even the length of the list changed: the same prompt produced "as few as 2-3 recommendations, and equally often 10 or more". The order was the least stable part. SparkToro estimates it takes about 1,000 runs before two lists come back in the same order. A brand that sits fourth in today's screenshot may lead tomorrow's answer, or be missing from it.

Where AI answer variability comes from

Sampling. A language model picks each word among several likely ones, so two runs can drift apart.
The serving system. Even with sampling set to always pick the likeliest word (temperature 0), outputs vary. Thinking Machines Lab asked an open model "Tell me about Richard Feynman" 1,000 times at temperature 0 and got 80 different completions. All matched for the first 102 tokens, then split between "Queens, New York" and "New York City". The lab traces the cause to server load, which changes how requests are batched.
Retrieval. Engines that search the web may run different searches and read different pages from one run to the next.
Wording and language. Chen and colleagues found that rewording a prompt changes the cited domains more than the brands named, and that switching language moves results more than rewording does.

Example: the same prompt, three runs

As an illustration, ask Gemini "Which CRM should a five-person real estate agency use?" three times in one day. The first answer lists five CRMs. The second lists three, one of them new. The third opens with a different tool and cites a different review site. None of these answers is wrong, and none of them is "the" answer: together they show which brands Gemini tends to consider for that need.

How to measure through the noise

Repeat each prompt on a schedule instead of checking it once, which is what prompt tracking does.
Track rates, not ranks. The share of answers that name you, your mention rate, holds steadier than any single position.
Compare with competitors on the same runs, so a change in the engine affects every brand alike.
Read the trend over several weeks before acting on a change.

Hawkscan runs each tracked prompt every day on ChatGPT, Microsoft Copilot, Gemini and Google AI Mode and shows the full answers of each run, word for word, so you can see what changed from one day to the next. The AI visibility score then sums those runs up in one number.