The most-quoted drift statistic says something narrower than you think
You will find this line across a dozen blogs: 91% of ML models fail due to drift.
The underlying research says something more precise and more interesting. A peer-reviewed study in Nature's Scientific Reports tested 128 model–dataset combinations across healthcare, finance, transportation and weather, and found temporal degradation in 91% of them. Not that 91% of production models fail. That across a broad test of model–dataset pairs, nearly all of them got worse over time.
That's not a smaller claim, it's a different one — and the difference matters. "Models fail" suggests an event you'd notice. "Temporal degradation" describes a process you won't, unless you're measuring for it.
Which is the real subject of this article. The gap between a notebook and production isn't primarily about tooling or deployment. It's that a notebook tells you a model worked once, on data you had, at a moment in time — and production is the business of finding out whether that's still true, continuously, for as long as the model runs.
Why production ML fails differently from production software
Traditional software fails loudly. An exception is thrown, a status code goes red, an alert fires, someone gets paged.
A degraded model does none of that. It keeps returning well-formed predictions at normal latency with a healthy error rate. The dashboard is green. The predictions are wrong.
There's a scenario ML engineers recognise immediately: a fraud detection model passing every health check — latency, throughput, error rate all normal — while fraudulent transactions slipped through at twice the usual rate. Drift had been eroding accuracy for weeks, entirely invisible to conventional monitoring.
That's the defining property of production ML. IBM notes accuracy can begin degrading within days of deployment. McKinsey found 40% of companies experienced noticeable AI performance degradation within the first year, tied to drift going unaddressed.
So the first principle of moving beyond experimentation is this: your existing monitoring stack cannot see the failure mode that matters most. Uptime, latency and error rate tell you the service is running. They tell you nothing about whether it's right.
What actually breaks when a notebook meets production
Notebooks are excellent for what they're for. The problems appear when their assumptions are carried forward.
Hidden state and execution order. A notebook's output depends on which cells were run, in what order, and what's still in memory. A notebook that produces a good model may not produce it again from a clean kernel — and if it can't, you cannot reproduce your own result, let alone audit it.
Data that isn't versioned. The code is in Git. The data usually isn't. Which means "the model that scored 0.94" is an unreproducible claim, because the thing it was trained on has since changed.
Manual steps nobody wrote down. The cell someone skipped. The file copied by hand. The parameter tweaked and not committed. These survive as tribal knowledge until the person leaves.
Training/serving skew. Features computed one way in the notebook with pandas, another way in the serving path in SQL or streaming. The gap is subtle, silent, and one of the most common causes of a model performing worse in production than in testing.
No concept of a "release." In a notebook, the model is a variable. In production, a model is an artefact with a version, a lineage, a promotion decision and a rollback path.
Worth keeping in proportion: training is a minority of the total effort in a production ML system — one practitioner estimate puts it around 15%. The other 85% is everything described above, plus everything below. Teams that budget for the 15% and are surprised by the rest are the ones whose models never leave pilot.
Pick a maturity level deliberately — probably not the highest one
The widely-used maturity model runs roughly:
- Level 0 — manual. Notebook to deployment by hand. Retraining is a project.
- Level 1 — automated training pipeline. Training runs on a trigger; deployment may still be manual.
- Level 2 — automated training and deployment, with CI/CD for the pipeline itself.
Most content treats this as a ladder you should climb to the top. That advice is wrong for most teams.
Level 2 pays for itself when you're deploying models frequently, running many models, or operating in a fast-drifting domain. Full CI/CD for pipelines is real engineering with real maintenance, and if you have three models that retrain quarterly, you'll spend more maintaining the automation than the automation saves.
The useful question isn't "what level are we?" It's "what is our retraining frequency, and what does an hour of manual deployment cost us?" Two models retrained twice a year do not justify the same infrastructure as forty models retrained weekly.
Target Level 1 well before reaching for Level 2. An automated, reproducible training pipeline with a model registry and real monitoring solves the majority of the pain. The remaining automation is an optimisation, not a prerequisite.
Build it in this order
- Reproducibility first. Before pipelines, before platforms: can you recreate any past training run exactly? Version code, data, configuration and environment together. If the answer is no, everything built on top inherits that uncertainty.
- A pipeline, not a script. Discrete stages — ingest, validate, transform, train, evaluate, register — that can be run, re-run and inspected independently. Orchestration tooling matters less than the decomposition.
- A model registry. Models as versioned artefacts with lineage: which data, which code, which parameters, which metrics, who approved promotion. This is also the single most useful thing you can hand an auditor, which is why it does double duty as governance evidence.
- Gates that can fail the pipeline. Automation without gating is just faster failure. At minimum:
- A data validation gate before training — schema conformance, feature distributions, null rates — that halts the run if data quality drops below defined thresholds.
- A model evaluation gate comparing the candidate against the current production champion on a held-out set, promoting only if it meets or beats the baseline.
- For regulated or sensitive use cases, bias and explainability checks generated and logged before any deployment.
The gate that matters most is the second one. Without an explicit champion–challenger comparison, "the new model is better" is an assertion, and teams ship regressions with complete confidence.
- Monitoring that watches the right things. Four layers, and most teams build only the first:
- System — latency, throughput, errors. Necessary, insufficient.
- Data — input distributions against the training baseline. Statistical tests like Kolmogorov–Smirnov, or Population Stability Index, where values above roughly 0.25 conventionally signal concerning drift.
- Prediction — output distribution shifts, which often move before you have ground truth.
- Outcome — actual accuracy once labels arrive. The truth, and usually the most delayed signal.
The distinction worth internalising: data drift means the inputs changed; concept drift means the world changed. Different causes, different fixes. Retraining resolves the first. The second may mean the model's framing is now wrong.
- Retraining triggers, not retraining schedules. Calendar-based retraining is either too frequent or too late. Event-based triggers — drift beyond a threshold, performance below a floor — react to what's actually happening. Drift speed is domain-dependent: financial models can drift within hours, healthcare models over months. Pick your trigger from your domain, not from a template.
- Rollback before you need it. Shadow deployment or canary release, and a tested path back to the previous version. A model you cannot roll back is a model you cannot safely update.
The 2026 addition: everything above, plus what LLMs break
If your production system includes foundation models, retrieval or agents, the classical picture is necessary but incomplete. Several drift modes have no equivalent in traditional ML:
Upstream model drift. If you build on a hosted model API, the model can change beneath you. Providers ship safety tuning, alignment adjustments and performance improvements, not always with a changelog. Your code is untouched; your behaviour isn't. This is the inverse of the classical setup — the model drifts under the application rather than the data drifting under the model — and almost no team monitors for it.
Retrieval drift. In RAG systems, the knowledge base, the embeddings and the index each drift independently. A stale corpus or a swapped embedding model degrades output quality while the LLM itself is unchanged. Retrieval quality has to be measured as its own problem.
Behavioural drift. Hallucination rate, refusal rate, verbosity, tone. These shift as a compound effect of the above and typically surface first in user complaints rather than in any metric you're tracking.
Prompts are code. Version them in Git with semantic versioning, test them, and record which prompt version produced which output. A prompt change is a release, and treating it as a config tweak is how untraceable regressions happen.
The practical consequence: your evaluation set becomes the load-bearing artefact. For classical ML, a held-out test set plus accuracy metrics carries most of the weight. For LLM systems, you need a scored set of real cases with known-correct outcomes, run against every model version, prompt version and retrieval change — because there is no accuracy metric to fall back on and the failure modes are qualitative.
One discipline worth adopting: isolate variables. Changing the model and the prompt simultaneously is the highest-risk change you can make, because when quality moves you can't attribute it. Change one, evaluate, then the other.
What not to build
The failure mode opposite to under-engineering is real, and expensive.
Don't buy a platform before you have the problem. End-to-end MLOps platforms make sense at scale. With two models and one data scientist, you'll spend the first quarter configuring a system to manage complexity you don't have.
Don't automate retraining you can't evaluate. Automated retraining without an evaluation gate is a machine for silently deploying worse models on a schedule.
Don't monitor what you won't act on. Every alert with no defined response trains the team to ignore alerts, including the ones that matter.
Don't build the pipeline before the model is worth running. MLOps infrastructure around a model with no demonstrated business value is expensive scaffolding around a question nobody answered.
Where to start this week
If you have models in production and none of this in place, in strict order:
- Take one production model and try to reproduce its training run from a clean environment. Whatever breaks is your first task — this exercise usually finds more than an audit would.
- Build the evaluation set. Real cases, known-correct outcomes, scored. It's the artefact that serves quality, debugging, promotion decisions and compliance simultaneously.
- Add the champion–challenger gate. Nothing promotes without beating the incumbent.
- Instrument input distributions against the training baseline, and set one threshold you'll actually respond to.
- Write down the rollback procedure and test it once.
None of that requires a platform, a migration, or a quarter of engineering time. It requires deciding that "it worked in the notebook" is the beginning of the work rather than the end of it.
Sources: Vela, Nalisnick et al., "Temporal quality degradation in AI models," Scientific Reports (2022); IBM on model drift; McKinsey on AI performance degradation; MLflow on pipeline validation and promotion gates.
Got models drifting quietly in production? Get in touch.
