TL;DR
- Every decision in an AI system has a half-life. Model choice and prompt syntax decay in months. Evaluation methodology and data quality standards compound for years.
- Models have dialects. A system prompt tuned for one model does not transfer to another, and newer models are often more capable and less forgiving at the same time.
- The abstraction layer is the tempting wrong answer. Teams who built one in 2023 traded model lock-in for framework lock-in and are in the same pain today.
- Evals are the real migration strategy. With a proper suite you know within hours whether a new model works. Without one, no architecture saves you.
The real question is whether you built AI in a way that makes the next migration cheap
Some decisions are one-way doors: hard to reverse, worth agonizing over. Others are two-way doors: easy to walk back through, worth making quickly and correcting as you learn.
A product that migrated three times in two years
Tursio is an enterprise analytics search product. In 2023, they built it on GPT-4-32k. It worked beautifully. Then OpenAI deprecated that model. They migrated to GPT-4.5-preview, which was deprecated three months later. Now they run on GPT-4.1, which interprets prompts so literally that their carefully tuned system broke in ways nobody noticed at first. Their regression pass rate dropped from 100% to 97%, which meant search queries that once surfaced the right answer now quietly returned the wrong one. (arxiv, July 2025)
Three percent sounds small. When your enterprise customers depend on search accuracy to find the right contract clause, the right policy document, the right compliance record, it is a crisis. And here is the part that should unsettle any leader investing in AI: Tursio picked the best available model each time. The problem was not their choice. The problem was structural.
We are seeing teams that went deep on specific models in late 2023 and 2024 now being punished for having been early. Worse, some of these teams are taking their carefully-tuned prompts, running them on a completely different model, and declaring the new model to be hype when the results disappoint. That is like switching from Spanish to Portuguese and blaming the language when your instructions are misunderstood.
But let us be honest about the other side of this. Early adopters also learned faster, shipped first, and won customers while others were still running evaluations. The cost of prompt migration is real but finite. We have seen migrations take anywhere from two to six weeks of engineering time depending on complexity. The cost of having built nothing while waiting for the ecosystem to stabilize is a different kind of loss entirely. The real question is whether you built in a way that makes the next migration cheap.
Models have dialects
If you have ever managed a team across geographies, you know that the same instruction lands differently depending on who receives it. Some people need explicit step-by-step scaffolding. Others take a brief and run with it. Give the brief-and-run person a detailed checklist and they will ignore half of it. Give the scaffolding person a vague brief and they will fill in the gaps incorrectly.
AI models work the same way. Each model family has developed its own prompting personality. Some (like GPT-4.1) interpret instructions hyper-literally. Others like Claude infer intent generously. Google's Gemini tends verbose and needs firm constraints to stay concise. OpenAI's own prompting guide acknowledges that prompt migration is likely required when moving between model versions within the same family, let alone across providers.
A five-page system prompt tuned for one model does not transfer to another. The Tursio team put it well in their paper: newer models are getting smarter in benchmarks but less flexible. More capable but less forgiving. They do exactly what you tell them, which means you had better say exactly the right thing, for that specific model.
The Tursio team made a sharp observation in their paper: newer models are getting smarter in benchmarks but dumber in flexibility. More capable but less forgiving. They do exactly what you tell them, which means you had better say exactly the right thing, for that specific model.
This is where the emerging discipline of prompt management enters. Prompt engineering is getting a prompt to work. Prompt management is keeping it working as models change underneath you. Most teams have invested heavily in the former. Very few have started on the latter.
The tempting wrong answer
When you see this problem clearly, the instinct is to reach for an abstraction layer. Build a model-agnostic interface. Swap providers like changing a battery.
It sounds right. It is how we solved vendor lock-in in every previous era of enterprise software. But LLMs are not databases or cloud providers. The interface to an LLM is natural language. A traditional software abstraction translates your intent into machine code, which is useful because machine code is hard to write. An LLM abstraction translates your intent into human language, through an intermediate framework. You have added complexity to reach a destination that was already in plain English. The same instinct applied to a whole platform, and why it fails the same way, is the subject of Enterprise Grade is Dried Spit With A Budget.
The results speak for themselves. Teams that built elaborate abstraction frameworks in 2023 are in just as much pain today as teams that hardcoded their model calls directly. They traded model lock-in for framework lock-in. In a domain this young, you cannot correctly anticipate what to abstract. An abstraction layer is only useful if you have guessed right about the axes of future change. Right now, nobody is guessing right consistently.
What actually works
Jeff Bezos has a useful framework for thinking about decisions under uncertainty. Some decisions are one-way doors: hard to reverse, worth agonizing over. Others are two-way doors: easy to walk back through, worth making quickly and correcting as you learn.
Model choice is a two-way door. So is your prompt syntax. So is your framework. These will change, probably more than once. The goal is not to prevent that change but to make it cheap.
Your evaluation methodology, on the other hand, is a one-way door. So are your data quality standards and your deep understanding of what “good” looks like for your specific use case. These are the decisions worth agonizing over because they compound over time and they are expensive to rebuild.
This is where evals become the real migration strategy. If you have a robust evaluation suite, one that is task-specific, tied to real failure modes, and validated against human judgment, you can swap models and know within hours whether the new one works. Without evals, no amount of architecture saves you. You are flying blind on every migration.
The Tursio team learned this the hard way. They built what they call a "migration testbed": queries categorized by complexity, edge cases that had failed before, automated regression checks across model versions. What had been a months-long trial-and-error process became a repeatable multi-week cycle. The testbed, and not any abstraction layer, is what made their third migration tolerable. Why measuring the wrong thing makes this worse rather than better is covered in Goodhart's Law and wrong measures for agent outputs.
The half-life question
Every decision in an AI system has a half-life. Some are long: your evaluation methodology, your data quality practices, your domain understanding, your team's accumulated intuition about what good looks like. Some are shockingly short: your model choice, your prompt syntax, your framework dependencies.
The teams that will thrive over the next two years are the ones that have developed an intuition for which of their decisions are decaying and which are compounding. For leaders, that means knowing which decisions to protect from churn and which to let your team swap freely.
Hold your strong architecture decisions loosely, yes. But hold them loosely in the right places. The skill is knowing the difference between what to grip and what to hold with open hands.
Enjoy your Sunday!
When Bharath Moro, Co-founder and Head of Products at Moative, explains guardrails using a salt shaker and tablecloth, guests usually take notes on their napkins.
Where Moative fits
Moative builds the evaluation suite before it builds the system, because that is the asset that survives the model churn underneath it. For a midmarket company this matters more than it does for a lab: you do not have an engineering team sitting idle waiting to redo a migration every nine months, so the cost of an unmeasurable system lands directly on operations. In healthcare revenue cycle management the failure mode is the quiet one described above, where accuracy slips by a few points and nobody notices until a payer does.
Frequently asked questions
What is the half-life of an AI system?
It is the rate at which each decision inside it decays. Model choice and prompt syntax have half-lives measured in months, because providers deprecate versions and successor models interpret instructions differently. Evaluation methodology, data quality standards and domain understanding compound instead of decaying.
Why do prompts break when you change the model?
Because each model family interprets instructions differently. Some follow wording literally, others infer intent, others need explicit constraints on length or format. A prompt is tuned to one model's reading habits, so moving it to another model changes what the instruction actually means.
Should I build an abstraction layer so I can swap AI models?
Usually not. The interface to a model is already natural language, so an abstraction adds a translation step without removing the underlying problem. Teams that built frameworks in 2023 mostly traded model lock-in for framework lock-in, which is harder to leave.
How long does an LLM migration take?
In the cases we have seen, roughly two to six weeks of engineering time depending on complexity. With a proper evaluation suite in place it compresses sharply, because you find out within hours whether the new model passes rather than discovering failures in production over weeks.
What is prompt management?
Prompt engineering is getting a prompt to work. Prompt management is keeping it working as models change underneath you: versioning prompts, testing them against a regression suite, and knowing which ones are tied to which model behavior.
What is a migration testbed?
A standing set of test queries categorized by complexity, plus the edge cases that have failed before, plus automated regression checks that run against any new model version. It converts migration from a trial-and-error project into a repeatable cycle.
