10.2 Experiment tracking

What this is and why it exists

Three weeks after the run that produced your best number, somebody asks what produced it. Without a record the honest answer is that you do not know — the code has moved on, the settings were typed at a prompt, and the number lives in a screenshot. Experiment tracking makes that question answerable by recording every run with its settings, its results and its outputs, and the recording has to include the failures or the record misleads you.

The vocabulary

• Run — one execution, with its settings, results and outputs.
• Parameter — a setting that defined the run.
• Metric — a number the run produced, possibly at each step.
• Artefact — a file the run left behind: weights, plots, predictions, the configuration.
• Experiment — a named group of related runs.
• Sweep — many runs over a range of settings.
• Registry — the collection of models considered good enough to use.
• Stage — a marker on a registered model saying what it is for.

The mental model

The recording tool is an interface for logging and a place to look at what was logged. The reference describes it as "an API and UI for logging parameters, code versions, metrics, and output files when running your machine learning code and for later visualizing the results", where each run records metadata — "various information about your run such as metrics, parameters, start and end times" — and artefacts, "output files from the run such as model weights, images, etc". Two lines added to a training script and every run is recorded from then on.

What to log is where people go wrong, and the failure is always recording too little, discovered too late. The rule that works: log everything cheap, because storage is cheap and re-running is not.

Concretely, four groups. Parameters: the entire resolved configuration, not the two settings you were varying — the third one you were not thinking about is the one that turns out to matter. Metrics: the headline number and the breakdowns, logged per step where training is iterative so a curve exists rather than an endpoint. Artefacts: the model, the configuration file, the evaluation report, a sample of predictions, and the plots. And provenance: the commit hash, whether the working tree was clean, the data version, the library versions, the hardware, and who ran it.

The commit hash and the clean-tree flag together are the highest-value two fields in the whole record, because a run from uncommitted code is a run nobody can reproduce, and knowing that at the time is much better than discovering it during a review.

Comparison is where the recording pays back. Once runs are recorded uniformly, the interface does what a spreadsheet of screenshots cannot: sort by a metric, filter by a parameter, plot curves side by side, show which settings differ between two runs. That last is the one you will use most — take the run that worked and the one that did not and ask what was different, and frequently the answer is something neither of you would have guessed to write down.

Sweeps come nearly free once runs are recorded, since a sweep is many runs with settings drawn from ranges, and the tool that records runs can usually launch them and collect the results. Everything the hyperparameter topic said applies: prefer random or adaptive search over a grid, put learning rates and regularisation on a log scale, and treat a best value at the edge of a range as a range that was too narrow.

The trap is selective logging, and it is worth being precise about why it misleads. Logging only the runs that looked promising produces a record in which every run improved something, and you can no longer answer which change caused an improvement — because the runs that would have shown a change doing nothing are missing. It also hides the variance: five runs of the same configuration differ, and if only the best was recorded, your reported improvement may be entirely within that spread.

So record everything, including the crashes, the misconfigured runs and the ones you abandoned after two minutes. Tag them rather than deleting them. A failed run costs nothing to keep and answers a question later — usually "did we already try that?", which is asked constantly and answered wrongly from memory.

The registry is the boundary between experimentation and production, and it is the part that makes this operational rather than merely tidy. Any number of runs happen; a small number of models are ever considered for use. Registering a model gives it a name, a version and a stage, and the stage says what it is for — a candidate, the one currently serving, one retired.

Three things a registry buys. Serving code refers to a name and a stage rather than a file path, so promoting a model is a metadata change rather than a deployment. The lineage is preserved: the registered version points back to the run, and the run points back to the code and the data, which is exactly the trace a compliance review asks for and the next topic completes. And promotion becomes an event with a decision attached, rather than somebody copying a file to a server — which is what the release-gate topic builds on.

One habit that costs nothing and saves an afternoon regularly. Log the evaluation report as an artefact, not only the headline metric. Six weeks later the question is rarely "what was the accuracy" — it is "was it worse on the rare class", "how did it do on last quarter's data", "what did it get wrong", and the report answers those while a single number does not.

What you should now be able to explain or do

Say what a tracking tool records and add it to a training script. Log the four groups, including the entire resolved configuration and the commit hash with a clean-tree flag. Compare runs by parameter difference to diagnose a regression. Run a sweep using the hyperparameter rules. Explain precisely how selective logging misleads, and keep failed runs tagged. Register a model with a name, version and stage, and say what the registry buys. Log the evaluation report rather than only the headline number.

Check yourself