Why do machine-learning models fail silently in production?

Because a broken model does not crash — it keeps returning well-formed predictions that happen to be wrong. There is no exception, no 500, no stack trace; the API is up, latency is flat, and the output is a plausible-looking number. The failure lives in the gap between the world the model was trained on and the world it is now scoring, and that gap is invisible to ordinary monitoring. The ML Test Score paper makes the point about upstream changes directly: a source-system change "can radically change the feature’s meaning and thus confuse the model’s training or inference, without necessarily producing values that are strange enough to trigger other monitoring." So you cannot wait for an error — you have to monitor the statistics of the inputs and the outputs and alert when they drift, because nothing else will tell you.

What is the difference between data drift and concept drift?

Data drift (covariate shift) is when the distribution of the inputs changes but the relationship between inputs and the correct answer stays the same — a new user segment arrives, a sensor recalibrates, traffic shifts to mobile. The model is now seeing feature values it saw rarely in training, so its accuracy sags even though the underlying rule is unchanged. Concept drift is worse: the inputs may look identical, but the mapping from input to label has changed — fraud tactics evolve, buying behavior shifts after a price change, the meaning of "normal" moves. Data drift you can often catch by watching the input distributions alone; concept drift is invisible in the inputs and only shows up when you can measure prediction quality against ground truth, which is exactly the signal that arrives late or never.

How do you detect data drift with PSI and KL divergence?

You compare the live serving distribution of a feature against its training distribution, bin both, and compute a divergence. The Population Stability Index is PSI = Σ (aᵢ − eᵢ) · ln(aᵢ / eᵢ) over bins, where eᵢ and aᵢ are the expected (training) and actual (serving) proportions in bin i; the common rule of thumb is PSI < 0.1 means no significant change, 0.1–0.2 is moderate, and ≥ 0.2 is significant. KL divergence, Σ aᵢ · ln(aᵢ / eᵢ), measures the same gap asymmetrically. They are related: PSI is exactly KL(actual‖expected) + KL(expected‖actual), the symmetric sum, so PSI grows about twice as fast as one-directional KL and trips a fixed threshold at a smaller shift. Evidently’s defaults reflect this — a 0.1 threshold for PSI, KL, and Jensen-Shannon, a Wasserstein or K-S test on larger samples — and dataset drift is flagged when a set share of columns (default 0.5) drift at once.

What is training-serving skew and how do you monitor for it?

Training-serving skew is when the feature a model was trained on and the feature it is scored on are computed by two different codepaths that quietly disagree. The offline pipeline computes a "7-day click count" one way in a batch job; the online request path recomputes it another way under a latency budget; a rounding rule, a time window, or a null-handling difference makes the two diverge, and the model degrades even though nothing errored. The ML Test Score paper names it: "The codepaths that actually generate input features may differ at training and inference time … This is sometimes called ‘training/serving skew’ and requires careful monitoring to detect and avoid." You catch it by logging a sample of real serving traffic and comparing the serving feature values against the training values for the same examples, alerting on the number of skewed features and skewed examples. A feature store that computes each feature once from one definition removes the second codepath entirely.

When should you retrain a model — and when should you not?

Retrain when the signal says the model has decayed enough to matter: a drift metric has crossed its threshold, or measured prediction quality on live data has dropped below its bound, or the model has simply aged past the staleness you tested it can tolerate. Triggers are either scheduled (retrain every week) or drift-driven (retrain when PSI or a quality metric fires), and the safe path is always retrain → validate against a golden set → shadow the candidate on live traffic → canary → promote, with instant rollback to the last known-good model. But retraining is not free and not always right. If the drift is real but harmless to your task, if the new data is contaminated by an incident you have not diagnosed, or if a fresh model would just chase noise, retraining adds risk and cost without buying accuracy. The honest default is to make retraining a deliberate response to a measured regression, not a reflex on a cron schedule.

ML Reliability in Production

Q: What is training-serving skew and how do you monitor for it?

Training-serving skew is when the feature a model was trained on and the feature it is scored on are computed by two different codepaths that quietly disagree. The offline pipeline computes a "7-day click count" one way in a batch job; the online request path recomputes it another way under a latency budget; a rounding rule, a time window, or a null-handling difference makes the two diverge, and the model degrades even though nothing errored. The ML Test Score paper names it: "The codepaths that actually generate input features may differ at training and inference time … This is sometimes called ‘training/serving skew’ and requires careful monitoring to detect and avoid." You catch it by logging a sample of real serving traffic and comparing the serving feature values against the training values for the same examples, alerting on the number of skewed features and skewed examples. A feature store that computes each feature once from one definition removes the second codepath entirely.

Q: When should you retrain a model — and when should you not?

Retrain when the signal says the model has decayed enough to matter: a drift metric has crossed its threshold, or measured prediction quality on live data has dropped below its bound, or the model has simply aged past the staleness you tested it can tolerate. Triggers are either scheduled (retrain every week) or drift-driven (retrain when PSI or a quality metric fires), and the safe path is always retrain → validate against a golden set → shadow the candidate on live traffic → canary → promote, with instant rollback to the last known-good model. But retraining is not free and not always right. If the drift is real but harmless to your task, if the new data is contaminated by an incident you have not diagnosed, or if a fresh model would just chase noise, retraining adds risk and cost without buying accuracy. The honest default is to make retraining a deliberate response to a measured regression, not a reflex on a cron schedule.

When a web service breaks it throws a 500 and pages you; when a model breaks it keeps answering — confidently, wrongly, and silently. We name the four ways models rot (data drift, concept drift, upstream schema breaks, feature-pipeline rot), build a live drift simulator that computes PSI and KL off two real distributions and watches them trip their alarms at different speeds, then install the whole defense: monitoring the inputs and the outputs, evals on a schedule, the retrain-validate-shadow-promote loop, rollback when the model is stateful, and an incident runbook that triages bad model vs bad data vs bad code — drawn, computed, and animated.

System design · AI / ML. The source ↗

A free, interactive, animated visual explainer of ML Reliability in Production — built to be understood, not skimmed.

Questions

Why do machine-learning models fail silently in production?: Because a broken model does not crash — it keeps returning well-formed predictions that happen to be wrong. There is no exception, no 500, no stack trace; the API is up, latency is flat, and the output is a plausible-looking number. The failure lives in the gap between the world the model was trained on and the world it is now scoring, and that gap is invisible to ordinary monitoring. The ML Test Score paper makes the point about upstream changes directly: a source-system change "can radically change the feature’s meaning and thus confuse the model’s training or inference, without necessarily producing values that are strange enough to trigger other monitoring." So you cannot wait for an error — you have to monitor the statistics of the inputs and the outputs and alert when they drift, because nothing else will tell you.
What is the difference between data drift and concept drift?: Data drift (covariate shift) is when the distribution of the inputs changes but the relationship between inputs and the correct answer stays the same — a new user segment arrives, a sensor recalibrates, traffic shifts to mobile. The model is now seeing feature values it saw rarely in training, so its accuracy sags even though the underlying rule is unchanged. Concept drift is worse: the inputs may look identical, but the mapping from input to label has changed — fraud tactics evolve, buying behavior shifts after a price change, the meaning of "normal" moves. Data drift you can often catch by watching the input distributions alone; concept drift is invisible in the inputs and only shows up when you can measure prediction quality against ground truth, which is exactly the signal that arrives late or never.
How do you detect data drift with PSI and KL divergence?: You compare the live serving distribution of a feature against its training distribution, bin both, and compute a divergence. The Population Stability Index is PSI = Σ (aᵢ − eᵢ) · ln(aᵢ / eᵢ) over bins, where eᵢ and aᵢ are the expected (training) and actual (serving) proportions in bin i; the common rule of thumb is PSI < 0.1 means no significant change, 0.1–0.2 is moderate, and ≥ 0.2 is significant. KL divergence, Σ aᵢ · ln(aᵢ / eᵢ), measures the same gap asymmetrically. They are related: PSI is exactly KL(actual‖expected) + KL(expected‖actual), the symmetric sum, so PSI grows about twice as fast as one-directional KL and trips a fixed threshold at a smaller shift. Evidently’s defaults reflect this — a 0.1 threshold for PSI, KL, and Jensen-Shannon, a Wasserstein or K-S test on larger samples — and dataset drift is flagged when a set share of columns (default 0.5) drift at once.
What is training-serving skew and how do you monitor for it?: Training-serving skew is when the feature a model was trained on and the feature it is scored on are computed by two different codepaths that quietly disagree. The offline pipeline computes a "7-day click count" one way in a batch job; the online request path recomputes it another way under a latency budget; a rounding rule, a time window, or a null-handling difference makes the two diverge, and the model degrades even though nothing errored. The ML Test Score paper names it: "The codepaths that actually generate input features may differ at training and inference time … This is sometimes called ‘training/serving skew’ and requires careful monitoring to detect and avoid." You catch it by logging a sample of real serving traffic and comparing the serving feature values against the training values for the same examples, alerting on the number of skewed features and skewed examples. A feature store that computes each feature once from one definition removes the second codepath entirely.
When should you retrain a model — and when should you not?: Retrain when the signal says the model has decayed enough to matter: a drift metric has crossed its threshold, or measured prediction quality on live data has dropped below its bound, or the model has simply aged past the staleness you tested it can tolerate. Triggers are either scheduled (retrain every week) or drift-driven (retrain when PSI or a quality metric fires), and the safe path is always retrain → validate against a golden set → shadow the candidate on live traffic → canary → promote, with instant rollback to the last known-good model. But retraining is not free and not always right. If the drift is real but harmless to your task, if the new data is contaminated by an incident you have not diagnosed, or if a fresh model would just chase noise, retraining adds risk and cost without buying accuracy. The honest default is to make retraining a deliberate response to a measured regression, not a reflex on a cron schedule.

ML Reliability in Production

Questions

Related explainers