27 October 2026

How often do AI recommendations change?

Often. Ask ChatGPT, Perplexity or Google's AI the same buyer question twice and the list of brands it names can differ, sometimes a little, sometimes a lot. That is normal behaviour, not a glitch. It means a single answer, or a single screenshot, says very little about whether an assistant recommends a brand. What is meaningful is how often a brand is named across many askings of the same questions.

Why the same question gets different answers

There are three main reasons, and they stack:

Sampling. Language models choose each word from a range of likely options rather than always taking the single most likely one. Two runs of the same prompt can take different paths and end up naming different products.
Live search. When web search is on, the assistant reads pages at the moment of the question. Search results shift, pages are updated, and the set of pages it reads can differ between two askings minutes apart.
Model updates. Providers update their models regularly. An update can change what a model knows and how it phrases recommendations, sometimes overnight.

What tends to stay stable

Variation is not the same as randomness. A brand that is widely described across the web has many paths into an answer, so it tends to be named again and again. A niche or newer product has fewer paths, so it appears in some answers and not others.

That pattern is why repeated measurement works. The leaders show up again and again, and the brands at the edge show a rate somewhere in between, which is exactly the information a brand needs.

How to measure through the noise

A measurement that holds up has a few simple parts:

Use a fixed set of real buyer questions, not one, because phrasing changes the answer.
Ask each question several times on each assistant, in a fresh session each time.
Count the share of answers that name each brand, rather than its position in one answer.
Show an error bar. Being named in 6 of 30 answers is a range, not exactly 20%, and a small gap between two brands may not be a real difference.
Repeat on a schedule, so a change you see is a trend rather than one unusual day.

How Shortlist does it

Each Shortlist leaderboard asks 10 buyer questions, each 3 times, on five assistants: ChatGPT, Perplexity, Claude, Google AI Overview and Google AI Mode. That is 150 answers per category. Every brand gets a mention rate with a 95% Wilson interval, every answer is stored word for word and linked from the page, and the run is repeated every month.

As our leaderboards fill up, we will publish what the repeat runs show about how stable each category is. Until then, the safest rule is simple: never judge your AI visibility from one answer.

Questions

Is it a problem that AI answers change?

Not for measurement, as long as you ask many times and report a rate with an error bar. It is a problem only when a single answer is treated as the truth.

How many answers do I need?

Enough for the error bar to be useful. A few dozen answers per assistant gives a rough picture; more narrows the range. Shortlist uses 150 answers per category run.

Does turning off web search make answers stable?

It removes one source of change, but sampling still varies the answer, and buyers usually use assistants with search on. Measure the way buyers use them.