10.3 Data and model versioning

What this is and why it exists

Source control gives you a guarantee about code: any past state can be recovered exactly. It gives you nothing about the data, which is where most of the variation in a model actually comes from. Data and model versioning extends that guarantee to the things source control cannot hold, so a prediction can be traced back to the weights that produced it and the data that trained them — which is what a compliance review asks for, and what a debugging session needs.

The vocabulary

• Content hash — a fingerprint of a file's contents, identifying it exactly.
• Pointer file — a small committed file naming a large stored one by its hash.
• Remote storage — where the large files actually live.
• Dataset card — written documentation of what a dataset is and is not.
• Schema contract — a declared shape the data must conform to.
• Lineage — the chain from prediction to model to code to data.
• Reproducibility — getting the same result again, to a stated degree.

The mental model

The problem is that "the same pipeline" is not the same pipeline if the data changed. Two runs of identical code produce different models when the input differs, and upstream data changes constantly and quietly: a column is renamed, a source system starts sending nulls, a backfill rewrites last month, a filter is adjusted somewhere you do not own. Version only the code and you cannot even prove what changed, which turns a regression into an argument.

The mechanism is the same one source control uses, applied at a distance. Take the content hash of a data file, put a small pointer into the repository naming that hash, and keep the file itself in remote storage. The repository stays small, the pointer is versioned like any other file, and checking out an old commit gets you the pointer that fetches the exact data that commit used. Branching, switching and comparing all work because you are versioning identity rather than bytes.

Two tools are commonly used. One is source-control-shaped: pointers in the repository, a remote for the data, and a command to fetch what a commit needs — it fits smallest and it is the sensible default when data lives in files. The other is storage-shaped, giving an object store itself commits and separate lines of development so you can prepare a change and switch atomically — heavier, and right when many teams share one large store. The choice matters much less than doing it at all, and doing it badly is still far better than not doing it.

Dataset cards are the human half. A hash says which bytes; a card says what they mean. Write down what the dataset contains, where it came from and under what terms, when it was collected and over what period, how it was cleaned and what was dropped, what is known to be wrong or missing, which populations are under-represented, and what it should not be used for. Write it while you still remember, because six months later the person who knows has moved on and the answers are gone.

That last field is the one that earns its place. A dataset assembled for one purpose gets reused for another by somebody who did not assemble it, and the note saying "this excludes accounts closed before the migration" is what stops a model being trained on a population it will not see.

The schema contract is the machine half, and it is what catches an upstream change before it reaches a model. Declare the shape: which columns exist, their types, which may be null, permitted ranges, permitted category values, and expected relationships between columns. Check it at ingestion, and fail the pipeline when it is violated rather than warning into a log nobody reads.

The failure this prevents is specific and common. An upstream column changes from a code to a description, or its units change, or a category is added. Nothing errors — the values are still strings, the pipeline still runs, the model still trains — and quality degrades for reasons nobody can find, because the change happened in a system your team does not own and nobody announced it. A schema check turns a silent degradation into a loud failure at the boundary, which is the cheapest place to handle it.

Model versioning and lineage complete the chain, and the target is one question: given this prediction, what produced it? Answering it requires each link recorded. The prediction carries the model version that served it. The model version points to the run. The run records the commit and the data version. The data version resolves to exact content.

Record the model version with every prediction, in the response or in the log. It costs one field and it is the difference between "we changed something around then" and "predictions after this timestamp came from version four, which was trained on this data". A compliance review asks that question directly; so does every serious investigation of a bad outcome, and so, eventually, does a customer.

Then the limits, which must be stated rather than implied. Exact reproducibility is bounded by things you do not control. Accelerator operations accumulate in nondeterministic order, so identical inputs give bitwise different outputs. Library and driver versions change the arithmetic. Parallel data loading changes the order examples arrive in. Some hardware is not deterministic for some operations.

So state what you actually guarantee. "The same data, the same code and the same configuration, producing a model within this tolerance on this metric" is an honest and useful statement. "Bitwise identical" is achievable only with deliberate effort, at a real cost in speed, and only where every version is pinned. Overstating this is worse than naming the limit, because somebody will eventually check, and a statement that fails is worth less than a modest one that holds — which is the same honesty the reproducibility discussion in the deep learning module asked for, arriving now with a compliance reviewer attached.

What you should now be able to explain or do

Say why versioning code alone leaves a regression unprovable. Explain the pointer-and-hash mechanism and why branching works. Choose between the file-shaped and storage-shaped tools with a reason. Write a dataset card, including what the data should not be used for. Declare a schema contract and fail the pipeline at ingestion, and describe the silent failure it prevents. Record model version with every prediction and trace a full lineage chain. State a reproducibility guarantee that is true, and name what bounds it.

Check yourself