Why do AI engines give a different answer every time you ask?
AI engines answer from a probability distribution and from a fresh web fetch, so two runs of one prompt are two samples rather than one lookup performed twice. The variance is large and it has been measured: a University of St. Gallen preprint tracking four AI search engines daily for about 45 days found the sources cited on consecutive days overlapping by only 34% to 42%, and the brands named overlapping by 45% to 59%. Pinning the temperature to zero does not close it either: 1,000 identical temperature-zero requests in a Thinking Machines Lab test returned 80 different completions, because server batching changes the arithmetic. The practical consequences are simple. One check proves nothing, seven to ten runs per engine make a percentage meaningful, and the half of the system you can influence is which pages are eligible to be drawn at all.
Why do AI engines give a different answer every time you ask?
AI engines give a different answer each time because nothing in the pipeline is a lookup. A language model produces text by sampling the next token from a probability distribution, so two runs take two paths through the same set of likely words. Ask which vendors serve a market and the model is not reading a list, it is generating one, and a name sitting just below the cut on one run clears it on the next.
Retrieval adds a second, larger source of movement. An answer engine searches the live web before it writes, and the results it gets back depend on the index at that moment, on which sub-queries it happened to issue, and on how the reranker scored a set of pages that are themselves changing. A page published this morning can enter the candidate set this afternoon and drop out tomorrow without anyone touching it.
Then come the differences between you and the next person asking. Memory, custom instructions, account history, country, device and the model version in service that day all shift the output, which is why Similarweb's explainer on whether ChatGPT answers everyone the same way concludes that it does not. Your own account is the worst possible instrument for checking how an engine describes your company, because it has been reading your conversations for months.
None of this is a malfunction. Sampling is what makes the output read like language instead of a database dump, and the fresh fetch is what keeps the answer current. Both are deliberate, and both mean that any statement beginning "ChatGPT says" describes one draw from a distribution, not a fact about the engine.
How different are two AI answers to the same question?
Two answers to one question differ more than most people expect, and there is now a measured figure for it. The preprint Don't Measure Once: Measuring Visibility in AI Search, by Schulte, Bleeker and Kaufmann of the University of St. Gallen and published on arXiv in April 2026, tracked four AI search engines daily across four campaign verticals for roughly 45 days, from 24 January to 20 March 2026, using eight prompts per campaign.
Across consecutive days, the researchers found that the set of cited sources overlapped by only 34% to 42%. The set of brands named held together better at 45% to 59% overlap, which fits the intuition that an engine changes its evidence more readily than it changes its recommendation. Neither number is close to the stability a marketing dashboard implies when it draws a line between two weekly readings.
The study also re-ran prompts simultaneously, eight prompts per campaign sent up to ten times to each of the four engines in succession, in a design that removes the passage of time as an explanation. The cited sources still overlapped by only 32% to 43%, which is effectively the same spread as the readings taken a day apart. That is the important control, because it rules out the web changing underneath the engine. The engine itself lands somewhere different on two draws taken at the same instant.
| What was compared | Overlap | What it means for you |
|---|---|---|
| Cited sources, consecutive days | 34% to 42% | Most of the citation list is replaced day to day |
| Brands named, consecutive days | 45% to 59% | Recommendations are steadier than evidence, still volatile |
| Cited sources, simultaneous re-runs | 32% to 43% | The engine alone produces different draws at one moment in time |
Source: Schulte, Bleeker and Kaufmann, University of St. Gallen, arXiv preprint 2604.07585, April 2026.
Can you make an AI engine give the same answer twice?
Not reliably on a shared endpoint: the same prompt gives a different answer even with the temperature set to zero, for a reason that sits lower down the stack than most explanations of this reach. A temperature of zero makes the model take its most likely next token every time, which ought to make the output reproducible. Horace He and colleagues at Thinking Machines Lab tested whether it is, in Defeating Nondeterminism in LLM Inference, published 10 September 2025. They sent one prompt 1,000 times to Qwen3-235B-A22B-Instruct-2507 at temperature zero, 1,000 tokens per completion, and got back 80 distinct outputs, the most common of which appeared 78 times. Read at source on 28 October 2026.
Where the answers split is the instructive part. All 1,000 completions were identical for the first 102 tokens and first diverged at token 103, where 992 of them continued with "Queens, New York" and 8 with "New York City" (same study). One token resolved differently and everything after it was a different answer. The cause the authors identify is not the usual story about concurrency and floating point. The kernels that run inference are not batch invariant: the server batches your request with whatever other traffic arrived at that moment, the batch size changes the order the numbers are added up in, and floating-point addition is not associative, so the sum moves. In their words, the primary reason inference endpoints are nondeterministic is that "the load (and thus batch-size) nondeterministically varies." When they rebuilt the kernels to hold the reduction order fixed, all 1,000 completions came back identical.
Two things follow for anyone checking how an engine describes their company. Whether your answer repeats depends on other people's traffic, which is not a setting you or the vendor exposes, so a repeated prompt is a second sample even with everything pinned. And this is the smallest of the sources of movement on this page, not the largest: it took a hundred identical tokens before a fixed input diverged at all, while the retrieval layer above it replaces most of the citation list between runs. Mahe Chen and Nick Koudas, in What Drives AI Citations? Content, Retrieval and Model Knowledge in Generative Engine Optimization, arXiv 2610.15033, put which pages the engine's own search retrieves at about 32% of the variation in citations against 2.8% for page properties. So retrieval is the first place to look when an answer changes, not the model's sampling.
One limit on that measurement. It was run on an open-weight model and the researchers' own serving stack, not on ChatGPT, Gemini or Perplexity, and none of those vendors documents whether its kernels are batch invariant. What transfers is the mechanism, and the fact that a temperature of zero is not on its own a promise of the same answer twice.
Does that mean AI visibility tracking is pointless?
Variance does not make AI visibility tracking pointless, it makes single observations pointless. A brand that appears in nine runs out of ten and a brand that appears in one out of ten are in genuinely different commercial positions, and no amount of sampling noise erases that gap. The measurement that survives is a rate across many runs, reported with a range.
What does not survive is most of what gets circulated internally. A screenshot of one answer proves nothing. A rank position inside a single response is an artefact of one draw. A week-on-week movement built from one reading per week is noise plotted as a trend, and it will send a team chasing a change that never happened.
Sample design matters more than tool choice here. Fix the prompt set and keep it fixed, because changing the wording changes the question and makes the comparison meaningless. Run each prompt several times per engine in fresh sessions with memory disabled. Record which sources were cited as well as whether you were named, since the source list is where a fix shows up first, and score the overlap between one round’s source list and the next so you know how much churn the engine carries before you read anything into your own absence. Scoring citation stability between runs works that number through, along with the published turnover rates to read it against.
Report it honestly afterwards. A visibility figure of 60% across ten runs on four engines is a real statement about a distribution. The same figure derived from one run per engine is a coin toss dressed as a metric, and presenting it as the latter is how measurement programs lose credibility the first time somebody re-runs the prompt in the room. Our guide to tracking AI visibility sets out the four numbers worth keeping.
How many times should you run a prompt before you trust the number?
Run each prompt roughly seven to ten times per engine before trusting the number it produces. The St. Gallen researchers reported the standard error of an estimated per-brand detection rate dropping below 0.10 at seven runs and continuing to tighten beyond that, which is the point where a visibility percentage begins describing the engine's behaviour rather than the luck of a particular draw.
Seven runs is a floor for a single prompt on a single engine, not a budget for a program. Multiply by the number of prompts your buyers actually ask and by the engines you care about, and the volume is why this is automated rather than done by hand in a browser. It is also why a serious prompt set is short and deliberate: twenty questions that matter, sampled properly, beat two hundred sampled once.
Hold the conditions steady while you do it. Fresh sessions, memory and custom instructions off, the same geography each time, the same wording every run, and a timestamp on every observation. Any of those left loose will produce a movement in the data that has nothing to do with your pages, and the movement always seems to arrive the week before a reporting cycle.
Set your detection threshold before you look at results. Decide in advance what counts as being named, whether a citation without a mention counts, and how a competitor recommended in passing is scored. Those definitions decide the number more than the engine does, and writing them down is the difference between a measurement and an opinion. We cover the operational build in how to automate AI visibility reporting.
Can a tool tell you why your ChatGPT citations changed?
A tool can tell you that the citation set changed, and whether the change is bigger than your own noise. It cannot tell you why. No engine publishes how a retrieved page becomes a cited one: all four document crawler access and eligibility and then stop, which our guide to how AI engines choose which sources to cite sets out vendor by vendor. The step that changed is therefore not observable from outside, so any product that reports a cause is showing you an inference. That is not a reason to distrust the category. It is a reason to ask which column of the table below a given claim sits in.
| The claim | Establishable | From what |
|---|---|---|
| The cited sources for this prompt changed | Yes | Overlap between this round's cited URLs and the last round's, on the same prompt and the same engine |
| The change is larger than this engine's ordinary churn | Yes, with enough runs | Your own baseline rounds. Schulte, Bleeker and Kaufmann put the standard error of a detection rate below 0.10 at seven runs per prompt, which is the floor for saying anything at all |
| Your page is no longer being retrieved | Mostly | Absence from every answer rather than loss to a named rival, plus whether the search agents still fetch the URL in your server logs |
| Something else is being cited in your place | Yes | The sources the engine cited instead, read rather than counted |
| The engine changed how it chooses between sources | No | Not documented by any vendor. A tool can only observe the output of that step |
| This particular answer came out differently by chance | No, not after the fact | Sampling and batching leave no trace in the answer. It is only visible as spread across repeated runs |
So the useful question to put to a tool is not why, it is how much and against what. Ask it how many runs each number rests on, whether it keeps one series per engine, and what it holds as the baseline, because a change reported without those three is a draw presented as a finding. Why two AI visibility tools report different numbers for the same brand works through where the methods diverge, and scoring citation stability between runs gives the arithmetic for the first row of the table.
The why is still answerable, just not by the dashboard. Run the prompt, list the sources the engine cited instead of you, and open them. Newer pages answering the question you used to own is a depth and freshness problem. Pages answering a different question means the prompt has drifted and you need a section you do not have. Nothing in your place at all, with a shorter answer, means the engine found less to work with across the whole category. That reading is a person's job, and it is the first step in measuring AI citation decay rather than reacting to it.
What can you actually control when the answer is random?
What you control is eligibility, and eligibility is where most of the variance is decided. An engine can only vary its answer among the pages it managed to retrieve, parse and match to the question. A page that is not in that candidate set has a zero percent chance on every draw, and a page that is unambiguously the clearest answer to the exact question sits near the centre of the set rather than at its margin, where sampling noise does its damage.
Being at the centre is a writing problem before it is a technical one. Use the question the way a buyer says it out loud as the heading. Answer it outright in the first sentence underneath. Keep the section self-contained, because a model lifting it never saw the paragraph above. Name the source of every number in the same paragraph as the number, since an unsourced figure gives a model nothing to verify and reads as invented.
Corroboration steadies the rest. A company described consistently across its own site, the review and comparison pages in its category and the places its customers talk is easier for an engine to resolve into one entity, and easier to name with confidence on a run where the evidence came back thin. Third-party substantiation carries 15% of the score in the SIGNALS framework for exactly this reason.
Expect improvement to look like a rising floor rather than a fixed position. Pages that get the basics right move from being named occasionally to being named usually, and the day-to-day flicker continues throughout. That is the correct shape of success in a system that samples, and a program judged on any single answer will miss it entirely. Our AI citation checklist lists the page-level work in the order worth doing it.
What else do people ask about ChatGPT giving different answers?
Why does ChatGPT give a different answer every time you ask the same question?
Because a language model samples its next word from a probability distribution rather than looking up a stored answer, and because the web results it reads are re-fetched and re-ranked on every run. Memory, custom instructions, location and the model version in service that day push the output further apart. Two answers to one prompt are two draws from a distribution, not one fact retrieved twice.
How different are two AI answers to the same question?
Far enough apart that a single check tells you very little. The University of St. Gallen study Don't Measure Once, published on arXiv in April 2026, tracked four AI search engines daily for about 45 days and found that the sources cited on consecutive days overlapped by only 34% to 42%, with the set of brands named overlapping by 45% to 59%. The brand list is steadier than the source list, and neither is stable.
How many times should you run a prompt before you trust the result?
Around seven to ten runs per engine for each prompt you care about. The St. Gallen researchers reported the standard error of a brand's estimated detection rate falling below 0.10 at seven runs, which is the point where a visibility percentage starts describing the engine rather than the draw. Run them in fresh sessions with memory off, keep the prompt wording fixed, and report a range instead of a single number.
Does answer variance mean AI visibility tracking is pointless?
No, it means point estimates are pointless and distributions are not. A brand named in nine of ten runs and a brand named in one of ten are in genuinely different positions, and that difference survives the noise. What does not survive is a single screenshot, a rank position for one answer, or a week-on-week movement built from one observation per week.
What can you control if AI answers are random?
The retrieval half, which is where most of the variance lands. An engine can only vary its answer among pages it can reach, parse and match to the question, so being unambiguously the clearest page on that question raises your floor in every draw. Wording the question the way buyers word it, answering it in the first sentence and sourcing every claim moves you from the margin of the candidate set toward its centre.
What else do people ask about getting the same answer twice?
Can you make ChatGPT give the same answer twice?
Not reliably, and not by setting the temperature to zero. Thinking Machines Lab sent one prompt 1,000 times to an open-weight model at temperature zero, 1,000 tokens per completion, and got back 80 distinct outputs; the completions were identical for 102 tokens and then diverged. The cause is that inference kernels are not batch invariant, so the server batching your request with other traffic changes the order the arithmetic runs in and the result moves with it. Whether your answer repeats therefore depends on other people's traffic, which no setting exposes. Source: Horace He and colleagues, Defeating Nondeterminism in LLM Inference, 10 September 2025.
Can a tool tell you why your ChatGPT citations changed?
It can tell you that the cited sources changed and whether the change is bigger than your own baseline churn. It cannot tell you why, because no engine documents how a retrieved page becomes a cited one, so any stated cause is an inference. What is establishable: that the set moved, that your page is no longer being retrieved at all, and which sources were cited in your place. What is not: that the engine changed how it weighs sources, or that one answer came out differently by chance. Ask a tool how many runs each number rests on, whether it keeps one series per engine, and what it treats as the baseline.