10.6 CI/CD for machine learning

What this is and why it exists

An ordinary pipeline asks whether the code is correct. A machine learning pipeline has to ask a second question the first cannot answer: is the model still good? The code can be entirely correct while the model regresses, and without a gate that measures the model, nobody notices until users do. This topic is that gate, the deployment strategies that give you warning, and the way back that has to have been rehearsed before the day it is needed.

The vocabulary

• Continuous integration — automated checks on every change.
• Continuous delivery — an automated path from a passing change to a deployment.
• Release gate — a check that must pass before something ships.
• Blue-green — two complete environments, with traffic switched between them.
• Canary — a small share of traffic sent to the new version first.
• Shadow — the new version run on real traffic without its output being served.
• Rollback — returning to the previous version.
• Kill switch — turning the feature off entirely.

The mental model

The ordinary checks come first and are frequently skipped in this field. Formatting, linting, type checking, unit tests, on every change, automatically. There is nothing machine-learning-specific about them and the excuse for omitting them — that this is research code — stops applying the moment somebody else has to run it.

What is worth adding for this field: fast tests that use tiny data, so the suite runs in under a minute. A training test on two rows checking the loss decreases. An inference test on a stub model checking the output shape and range. An evaluation test on invented predictions with a hand-computed answer. A test that the preprocessing produces identical output in the training path and the serving path — that last one is the training-serving skew check, and it belongs in the test suite rather than in a monitoring dashboard, because a test says so before the deployment rather than three weeks after.

Then the gate this topic exists for. A model change — new weights, new data, new features — must pass an automated evaluation before it can be released. The shape is simple: the pipeline loads the candidate model, runs it against the frozen evaluation set, compares against the current production model, and refuses to promote if it is worse.

Four rules make that gate real rather than decorative.

Gate on the metrics that matter, plural. A single headline number hides a model that improved overall and became much worse on the rare class that the feature exists for. Gate on the headline, on the per-segment breakdown, and on anything with a business meaning.

Compare against production, not against a threshold. A fixed threshold drifts out of relevance; "not worse than what is serving, by more than the noise" is the question you actually care about.

Allow for variance. Two runs differ, so a small drop is not evidence of a regression. Compare against the spread across repeated runs, and treat anything smaller as a tie — the same rule the tuning topic gave.

Make the evaluation set frozen and out of training. A gate that measures against data the model saw is not a gate. This is the golden set from the evaluation topics, versioned, in the repository, with a documented reason for every change to it.

And gate on more than quality: model size and inference latency belong there too. A model that is one point better and three times slower has failed a requirement nobody wrote down, and the gate is where you write it down.

The three deployment strategies differ in how much warning they give.

Blue-green keeps two complete environments and switches traffic from one to the other. The switch is instant and so is the way back, which is its whole value. It gives no warning, because everyone moves at once — but the reversal is a routing change rather than a redeployment.

Canary sends a small share of traffic to the new version, watches, and increases if the numbers hold. It gives real warning at real scale with a bounded blast radius, and it is the default for anything user-facing. It requires enough traffic for a small share to be meaningful and it needs the metrics per version, or you are watching an average that hides the thing you deployed.

Shadow runs the new version on real requests and does not serve its output — the current version answers, and the new one's results are recorded and compared. It is the only strategy with no user risk at all, which makes it right for a first release, for a model whose failure would be expensive, and for comparing two models on the true traffic distribution rather than on your test set. Its costs are that you pay for both versions and that it cannot measure anything downstream of the response, since nobody ever saw the output.

For a model change the sequence that works is: shadow to establish it behaves sanely on real traffic, then canary to see the effect on people, then full release.

Rollback and kill switches close the loop, and the point about both is rehearsal.

A rollback plan that has never been executed is a document, not a plan. Practise it — deliberately, on a normal day, with the people who would do it at three in the morning — and you will discover the things that are always discovered: the previous model artefact was cleaned up, the configuration is not versioned alongside the model, the database migration is not reversible, nobody knows who can approve it, or the deployment takes twenty minutes when the incident allows five.

A kill switch is the blunter and faster instrument: a flag that turns the feature off, without a deployment, without a rollback, without anybody's approval. It matters because sometimes the answer is not "the previous version" but "nothing, until we understand this". Keep the fallback behaviour defined — a rule-based default, a cached answer, a clear message — and test the switch on a schedule, because a switch nobody has flipped since it was written is a switch nobody knows works.

Two habits that make all of it usable. Deploy small changes often rather than large ones occasionally, because a small change that breaks something names its own cause. And record what shipped when — model version, code commit, data version, configuration, who approved it — because the first question in every incident is what changed, and the answer should be a lookup rather than an investigation.

What you should now be able to explain or do

Run the ordinary checks and add fast tests on tiny data, including a training-serving preprocessing equality test. Build an evaluation gate with the four rules, and gate on size and latency too. Choose among blue-green, canary and shadow by how much warning you need and what risk is acceptable. Sequence shadow then canary then full release for a model change. Rehearse a rollback and name what rehearsal typically uncovers. Keep a tested kill switch with defined fallback behaviour. Record what shipped when.

Check yourself