Why do AI engines give a different answer every time you ask?
AI engines answer from a probability distribution and from a fresh web fetch, so two runs of one prompt are two samples rather than one lookup performed twice. The variance is large and it has been measured: a University of St. Gallen preprint tracking four AI search engines daily for about 45 days found the sources cited on consecutive days overlapping by only 34% to 42%, and the brands named overlapping by 45% to 59%. The practical consequences are simple. One check proves nothing, seven to ten runs per engine make a percentage meaningful, and the half of the system you can influence is which pages are eligible to be drawn at all.
Why do AI engines give a different answer every time you ask?
AI engines give a different answer each time because nothing in the pipeline is a lookup. A language model produces text by sampling the next token from a probability distribution, so two runs take two paths through the same set of likely words. Ask which vendors serve a market and the model is not reading a list, it is generating one, and a name sitting just below the cut on one run clears it on the next.
Retrieval adds a second, larger source of movement. An answer engine searches the live web before it writes, and the results it gets back depend on the index at that moment, on which sub-queries it happened to issue, and on how the reranker scored a set of pages that are themselves changing. A page published this morning can enter the candidate set this afternoon and drop out tomorrow without anyone touching it.
Then come the differences between you and the next person asking. Memory, custom instructions, account history, country, device and the model version in service that day all shift the output, which is why Similarweb's explainer on whether ChatGPT answers everyone the same way concludes that it does not. Your own account is the worst possible instrument for checking how an engine describes your company, because it has been reading your conversations for months.
None of this is a malfunction. Sampling is what makes the output read like language instead of a database dump, and the fresh fetch is what keeps the answer current. Both are deliberate, and both mean that any statement beginning "ChatGPT says" describes one draw from a distribution, not a fact about the engine.
How different are two AI answers to the same question?
Two answers to one question differ more than most people expect, and there is now a measured figure for it. The preprint Don't Measure Once: Measuring Visibility in AI Search, by Schulte, Bleeker and Kaufmann of the University of St. Gallen and published on arXiv in April 2026, tracked four AI search engines daily across four campaign verticals for roughly 45 days, from 24 January to 20 March 2026, using eight prompts per campaign.
Across consecutive days, the researchers found that the set of cited sources overlapped by only 34% to 42%. The set of brands named held together better at 45% to 59% overlap, which fits the intuition that an engine changes its evidence more readily than it changes its recommendation. Neither number is close to the stability a marketing dashboard implies when it draws a line between two weekly readings.
The study also re-ran prompts simultaneously, eight prompts per campaign sent up to ten times to each of the four engines in succession, in a design that removes the passage of time as an explanation. The cited sources still overlapped by only 32% to 43%, which is effectively the same spread as the readings taken a day apart. That is the important control, because it rules out the web changing underneath the engine. The engine itself lands somewhere different on two draws taken at the same instant.
| What was compared | Overlap | What it means for you |
|---|---|---|
| Cited sources, consecutive days | 34% to 42% | Most of the citation list is replaced day to day |
| Brands named, consecutive days | 45% to 59% | Recommendations are steadier than evidence, still volatile |
| Cited sources, simultaneous re-runs | 32% to 43% | The engine alone produces different draws at one moment in time |
Source: Schulte, Bleeker and Kaufmann, University of St. Gallen, arXiv preprint 2604.07585, April 2026.
Does that mean AI visibility tracking is pointless?
Variance does not make AI visibility tracking pointless, it makes single observations pointless. A brand that appears in nine runs out of ten and a brand that appears in one out of ten are in genuinely different commercial positions, and no amount of sampling noise erases that gap. The measurement that survives is a rate across many runs, reported with a range.
What does not survive is most of what gets circulated internally. A screenshot of one answer proves nothing. A rank position inside a single response is an artefact of one draw. A week-on-week movement built from one reading per week is noise plotted as a trend, and it will send a team chasing a change that never happened.
Sample design matters more than tool choice here. Fix the prompt set and keep it fixed, because changing the wording changes the question and makes the comparison meaningless. Run each prompt several times per engine in fresh sessions with memory disabled. Record which sources were cited as well as whether you were named, since the source list is where a fix shows up first.
Report it honestly afterwards. A visibility figure of 60% across ten runs on four engines is a real statement about a distribution. The same figure derived from one run per engine is a coin toss dressed as a metric, and presenting it as the latter is how measurement programmes lose credibility the first time somebody re-runs the prompt in the room. Our guide to tracking AI visibility sets out the four numbers worth keeping.
How many times should you run a prompt before you trust the number?
Run each prompt roughly seven to ten times per engine before trusting the number it produces. The St. Gallen researchers reported the standard error of an estimated per-brand detection rate dropping below 0.10 at seven runs and continuing to tighten beyond that, which is the point where a visibility percentage begins describing the engine's behaviour rather than the luck of a particular draw.
Seven runs is a floor for a single prompt on a single engine, not a budget for a programme. Multiply by the number of prompts your buyers actually ask and by the engines you care about, and the volume is why this is automated rather than done by hand in a browser. It is also why a serious prompt set is short and deliberate: twenty questions that matter, sampled properly, beat two hundred sampled once.
Hold the conditions steady while you do it. Fresh sessions, memory and custom instructions off, the same geography each time, the same wording every run, and a timestamp on every observation. Any of those left loose will produce a movement in the data that has nothing to do with your pages, and the movement always seems to arrive the week before a reporting cycle.
Set your detection threshold before you look at results. Decide in advance what counts as being named, whether a citation without a mention counts, and how a competitor recommended in passing is scored. Those definitions decide the number more than the engine does, and writing them down is the difference between a measurement and an opinion. We cover the operational build in how to automate AI visibility reporting.
What can you actually control when the answer is random?
What you control is eligibility, and eligibility is where most of the variance is decided. An engine can only vary its answer among the pages it managed to retrieve, parse and match to the question. A page that is not in that candidate set has a zero percent chance on every draw, and a page that is unambiguously the clearest answer to the exact question sits near the centre of the set rather than at its margin, where sampling noise does its damage.
Being at the centre is a writing problem before it is a technical one. Use the question the way a buyer says it out loud as the heading. Answer it outright in the first sentence underneath. Keep the section self-contained, because a model lifting it never saw the paragraph above. Name the source of every number in the same paragraph as the number, since an unsourced figure gives a model nothing to verify and reads as invented.
Corroboration steadies the rest. A company described consistently across its own site, the review and comparison pages in its category and the places its customers talk is easier for an engine to resolve into one entity, and easier to name with confidence on a run where the evidence came back thin. Third-party substantiation carries 15% of the score in the SIGNALS framework for exactly this reason.
Expect improvement to look like a rising floor rather than a fixed position. Pages that get the basics right move from being named occasionally to being named usually, and the day-to-day flicker continues throughout. That is the correct shape of success in a system that samples, and a programme judged on any single answer will miss it entirely. Our AI citation checklist lists the page-level work in the order worth doing it.
What else do people ask about ChatGPT giving different answers?
Why does ChatGPT give a different answer every time you ask the same question?
Because a language model samples its next word from a probability distribution rather than looking up a stored answer, and because the web results it reads are re-fetched and re-ranked on every run. Memory, custom instructions, location and the model version in service that day push the output further apart. Two answers to one prompt are two draws from a distribution, not one fact retrieved twice.
How different are two AI answers to the same question?
Far enough apart that a single check tells you very little. The University of St. Gallen study Don't Measure Once, published on arXiv in April 2026, tracked four AI search engines daily for about 45 days and found that the sources cited on consecutive days overlapped by only 34% to 42%, with the set of brands named overlapping by 45% to 59%. The brand list is steadier than the source list, and neither is stable.
How many times should you run a prompt before you trust the result?
Around seven to ten runs per engine for each prompt you care about. The St. Gallen researchers reported the standard error of a brand's estimated detection rate falling below 0.10 at seven runs, which is the point where a visibility percentage starts describing the engine rather than the draw. Run them in fresh sessions with memory off, keep the prompt wording fixed, and report a range instead of a single number.
Does answer variance mean AI visibility tracking is pointless?
No, it means point estimates are pointless and distributions are not. A brand named in nine of ten runs and a brand named in one of ten are in genuinely different positions, and that difference survives the noise. What does not survive is a single screenshot, a rank position for one answer, or a week-on-week movement built from one observation per week.
What can you control if AI answers are random?
The retrieval half, which is where most of the variance lands. An engine can only vary its answer among pages it can reach, parse and match to the question, so being unambiguously the clearest page on that question raises your floor in every draw. Wording the question the way buyers word it, answering it in the first sentence and sourcing every claim moves you from the margin of the candidate set toward its centre.