TL;DR
- Ask the same model the same question twice, settings untouched, and you will often get two different answers. That is not your mistake and it is not a data leak.
- The cause is batching. How many requests the model is handling the instant yours lands changes the arithmetic slightly, and a nudged number becomes a different word.
- Agents almost always finish, which is the problem. They are quietly wrong in a way that looks exactly like being right, and most often on the messy case that was the reason you wanted help.
- Variance is highest where the model has most to work out on the spot. Tighten the task, clean the input, and the spread collapses. The fix is your data, not your pricing model.
Ask the same model the same question twice. Word for word, settings untouched, everything lined up so it should give you the same answer every time. You'll often get two different answers.
Why AI gives different answers to the same question
Most people assume that's their mistake. A stray setting, a typo, something they'll go fix. It isn't. The thing you're renting changes shape between one call and the next, and the reason is almost insultingly ordinary. That model is handling thousands of requests at the same time, and how many it happens to be juggling the instant yours lands depends on how busy it is. Nothing of yours mixes with anyone else's. No data leaks, nothing like that.
It's simpler and stranger than that. The machine is just doing a different amount of work each time, and that alone nudges a number somewhere deep in the math, and a nudged number becomes a different word, and a different word becomes a different answer. You didn't do a thing. It's a slightly different machine every time you knock.
I keep coming back to this, because of what it does to everything built on top.
This is not software, whatever the demo implies
We talk about AI as though it were software. Software is the same every time you run it. That's the whole miracle of it: write it once, trust it forever. This isn't that. This is closer to asking a very smart, very tired employee the same question on two different afternoons. Mostly you get the same answer. Sometimes you don't. And the times you don't are not announced.
That last part is the one that should worry you.
The agent always finishes
Sit through enough demos and you notice the agent always finishes. It always returns something, confident and done. Then you check the work. A good share of the time it's wrong, and not loudly wrong. It's quietly wrong, in a way that looks exactly like being right. The dashboard says complete. The status says resolved. The only way to catch it is to already know the answer. On the standard case it's fine. On the messy one, the patient who doesn't fit the template, the claim with the odd modifier, the edge that was the whole reason you wanted help, that's where it hands you the wrong thing with a straight face and moves on.
Why outcome pricing does not solve this
Now set that next to the pitch everyone's selling. Pay for outcomes.
It sounds fair. Pay for results, not software. I understand why an owner likes the sound of it. But you can't price a steady outcome on top of an unsteady process, and you can't even agree on whose work the outcome was. Did the AI clear the claim, or did your biller change the workflow last month? Two people will argue that invoice forever. The vendors know it. Watch closely this year and you'll catch them backing out of pure outcome deals, dressing them up again as service contracts, bolting on minimum fees, anything to put a floor under a number that won't hold still. We called this a non-starter over a year ago. The market is finding out the slow way.
The usual fixes don't fix it. A minimum fee plus a slice of the upside just splits the gamble between you and the vendor. The dice are the same dice.
So what actually narrows the swing?
What actually narrows the swing
The model wobbles most when it has the most to work out on the spot. A vague task over messy inputs, and it improvises, and improvisation is where the variance lives. Tighten the task and clean what you feed it, and the spread collapses. The same model that was a tossup over a pile of unstructured notes turns boringly reliable over a clean, well-shaped record. Same model. Different odds. The difference is the data, and the data is yours to fix.
This is why the order matters, and why we keep insisting on the unglamorous part. Own your data. Organize it. Shape it so the model barely has to guess. Do that first and outcomes stop being a marketing word and start being something you could actually stand behind, because the thing producing them finally holds still. Skip it, and no pricing scheme on earth saves you. You're only choosing how to split the cost of the wobble. The same argument, from the other end, is why the causes behind your operational problems have to be captured while the work happens rather than read out of the record afterwards, which I set out in Instrumenting Work to Avoid Expensive AI Rollouts.
The test you can run this week
You don't have to take anyone's word for any of this. You don't have to win an argument with a salesperson either. Run it yourself. Pull ten of your messiest cases, the ones that never go cleanly, and put each one through twice. Count how often the machine disagrees with itself. That's the number no demo will ever put on the screen. And it isn't really a verdict on the model. It's a reading of how much guessing your own data is forcing on it. Clean that up and the disagreements fade. Leave it as it is, and you're paying full price for a coin flip.
Where Moative fits
Moative works as an AI operating partner with mid-market companies, which in practice means we spend the first part of an engagement on the input side rather than the model side. We run the ten-messy-cases test on the process in question, use the disagreement rate as the baseline, then tighten the task boundaries and shape the data until the spread comes down. In healthcare revenue cycle management that is usually the difference between an agent that clears clean claims and one you can trust on the denials that actually cost you money. If a vendor is offering you outcome pricing on a process nobody has measured for variance, the test above is worth running before you sign.
Frequently asked questions
Why does AI give different answers to the same question?
Because the model is serving many requests at once, and the number it is handling when your request arrives changes the arithmetic very slightly. That small difference propagates into a different word choice and then a different answer. It is a property of how the systems are run, not a fault in your prompt or a sign that your data mixed with someone else's.
Does setting temperature to zero make AI deterministic?
It reduces deliberate randomness but does not remove the variation described here, because that variation comes from how the request is batched and processed rather than from a sampling setting. Two identical calls can still differ.
Is AI output variance a data security problem?
No. Nothing of your input mixes with anyone else's. The effect is arithmetic, not exposure.
Why do AI agents give confidently wrong answers?
Because they are built to return a completed result. An agent that cannot do the task well still finishes, and the output looks structurally identical to a correct one. There is no signal separating the two, which is why errors surface only when someone already knows the answer.
Should I agree to outcome-based pricing for AI?
Be careful. Outcome pricing on an unstable process transfers a gamble rather than removing it, and attribution is genuinely contested: neither side can prove whether the result came from the AI or from a change your own team made. Minimum fees and revenue shares split the gamble without shrinking it.
How do you make AI output more reliable?
Narrow the task so there is less to improvize, and clean the inputs so the model has less to guess. The variance is proportional to how much work the model has to do on the spot, which means the lever is your data rather than the model.
How do I test AI reliability on my own data?
Take ten of your hardest cases, run each one through twice, and count the disagreements. The rate tells you how much guessing your data is forcing, and it is the one number no vendor demo will show you.
