10.8 Monitoring, drift and retraining

What this is and why it exists

Production asks one question continuously: is the model still good? A service can return successful responses at excellent latency while its accuracy quietly rots, because nothing about a correct-looking response says the answer was right. This topic is the monitoring that notices, the two different kinds of drift and their two different responses, what to measure when the truth arrives weeks late, and when retraining is actually the answer.

The vocabulary

• Service metric — is the system responding, and how fast?
• Model metric — is the model still right?
• Data drift — the input distribution changing over time.
• Concept drift — the relationship between input and outcome changing.
• Label delay — the gap between a prediction and knowing whether it was right.
• Proxy metric — a stand-in measurable now that correlates with the truth later.
• Prediction drift — the distribution of the model's own outputs shifting.
• Retraining trigger — the condition that causes a retrain.

The mental model

Two layers of monitoring, and systems that watch only the first find out about the second from users.

Service metrics — availability, latency, error rate, throughput, saturation — say whether the system is up. They are necessary, they are what every alerting setup already covers, and they are entirely blind to model quality. A model that has become wrong serves successful responses quickly, so every service metric stays green while the thing degrades.

Model metrics say whether the answers are right. That requires knowing what the right answer was, which is where the interesting difficulty lives.

Start with what you can measure without any labels, because it is available immediately and catches a great deal.

Input distribution, feature by feature: the centre, the spread, the share missing, the set of categories seen. A feature that stops arriving, a category that appears for the first time, a value range that shifts — those are visible within a day and each of them is a real defect.

Prediction distribution: what the model is outputting. A classifier whose positive rate doubles overnight has either met a real change or broken, and both are worth waking up for. This is the cheapest early warning available and it is frequently the first sign of anything at all.

Confidence distribution: predictions becoming systematically less confident is a model meeting data it was not trained for.

Then the two kinds of drift, which need different responses and are commonly conflated.

Data drift is the input distribution moving while the underlying relationship holds. A new region, a new customer segment, a new device, a marketing campaign bringing a different population. The model may still be right for the inputs it understands and it is now seeing inputs it was not trained on. The response is more data covering the new region and a retrain.

Concept drift is the relationship between input and outcome changing. The same customer profile now behaves differently, the same symptoms now indicate something else, the same transaction pattern is no longer fraud because the fraud moved. The model is wrong about the world, and the old training data is now actively misleading — so the response is not more data but recent data, and possibly a re-examination of the features themselves.

Detecting them differs in kind. Data drift is detectable without labels, by comparing distributions between a reference period and now, using a statistical distance and alerting when it exceeds a threshold you set by watching normal variation first. Concept drift is not detectable from inputs at all — the inputs can look identical while the relationship has inverted — so it requires outcomes, which brings us to the hard part.

Delayed labels are the normal case, not an exception. Whether a loan defaults takes months. Whether a recommendation was good takes as long as it takes somebody to act. Whether a diagnosis was right takes a follow-up appointment. You cannot wait for the truth to notice a problem, so you need something measurable now.

The proxy metrics that work: user behaviour immediately after the prediction — accepted, ignored, overridden, corrected, escalated to a person; the rate of human override, which is the strongest single proxy in any system with a person in the path and is usually already being recorded for other reasons; downstream signals such as complaints, retries and abandonment; and the distribution measures above.

Two rules make proxies honest. Validate the proxy against the truth when it finally arrives — plot the proxy against the eventual outcome over a past period and confirm they move together, because an unvalidated proxy is a number you are trusting for no reason. And keep measuring the real metric on the delayed schedule anyway, so the proxy is an early warning and the truth is the verdict.

Retraining triggers, and the wasteful default worth naming. Retraining on a fixed schedule regardless of need is common and it is a poor default in both directions: it retrains when nothing changed, spending compute and introducing risk for no benefit, and it fails to retrain when something changed the week after the last run.

Better triggers, in the order to consider them. Performance-based: the real or proxy metric crosses a threshold. This is the right trigger and it requires the measurement above. Drift-based: the input distribution has moved beyond a bound — useful when performance is measured slowly, and it needs the caution that drift does not always mean degradation, since a model can be robust to a shift that looks large. Data-based: a meaningful volume of new labelled data has accumulated. Manual: somebody knows something changed — a product launch, a policy change, a new market — and this trigger is underrated, because a person frequently knows before any metric does.

And whatever the trigger, the retrained model goes through the release gate, because a retrain is a model change and a fresh model is not automatically a better one. Retraining on drifted data can bake in a temporary anomaly; retraining on data the current model influenced carries a feedback loop, the same one the recommender topic described. Automate the pipeline and keep the promotion decision gated, which is the arrangement that gets the speed without the risk.

Two habits that make monitoring real. Alert on things a person will act on, and route the rest to a dashboard — an alert nobody acts on trains everybody to ignore alerts. And log every prediction with its inputs, its output, the model version and a timestamp, because everything in this topic is computed from that log, and starting it after the incident means the incident cannot be investigated.

What you should now be able to explain or do

Distinguish service metrics from model metrics and say why the first are blind to quality. Monitor input, prediction and confidence distributions without labels. Distinguish data drift from concept drift by what changed, and give the different response to each. Say why concept drift cannot be detected from inputs. Choose proxy metrics under label delay, validate them against the truth, and keep the real metric on its own schedule. Choose a retraining trigger and say why the fixed schedule is wasteful in both directions. Gate every retrained model. Log every prediction from the first day.

Check yourself