10.5 Docker and reproducible environments

What this is and why it exists

"Works on my machine" is a statement about your machine, and it is the difference between code and a deployable thing. A container image freezes the environment — the operating system, the interpreter, the libraries, the accelerator toolkit — so the same image runs identically on your laptop, on the build machine and on the server. This topic is how to build one that is small and quick to rebuild, and the two failures particular to this field: bloated images and accelerator version mismatches.

The vocabulary

• Image — the frozen filesystem and configuration a container runs from.
• Layer — one step of the build, cached independently.
• Build cache — reuse of unchanged layers between builds.
• Multi-stage build — a build where later stages copy only what they need from earlier ones.
• Base image — the image your build starts from.
• Ignore file — the list of paths excluded from the build context.
• Compose file — a description of several services started together.

The mental model

An image is built in layers, and each instruction makes one. That matters because layers are cached: an unchanged layer is reused rather than rebuilt. The documentation states the rule and its consequence — "whenever a layer changes, that layer will need to be re-built", and "if a layer changes, all other layers that come after it are also affected", so "once a layer changes, then all downstream layers need to be rebuilt as well."

The whole art of a fast rebuild follows from that one sentence: put what changes rarely early, and what changes constantly late. Your dependency list changes weekly; your source changes every few minutes. So copy the dependency file and install dependencies first, then copy the source. Edit a source file and only the last layers rebuild, in seconds. Do it the other way round — copy everything, then install — and every one-character change reinstalls every dependency, which is a minutes-long rebuild you will do a hundred times.

Multi-stage builds keep the result small. They exist for anyone who has struggled to optimise a build file while keeping it readable and maintainable, and the mechanism is that "you can selectively copy artifacts from one stage to another, leaving behind everything you don't want in the final image". A first stage has the compilers, headers and build tools; the final stage starts clean and copies across only the installed environment and the application. The documentation's summary is the point: "the end result is a tiny production image with nothing but the binary inside. None of the build tools required to build the application are included in the resulting image."

For a Python service the shape is: install into a virtual environment in a build stage that has the compilers, then copy that environment and your source into a slim runtime stage. The final image contains what runs and nothing that built it, which is smaller to push and pull, faster to start, and has a smaller attack surface because a compiler that is not present cannot be used.

Then the bloat particular to this field, which is worth naming item by item. Model weights copied into the image — which makes the image enormous and, worse, means a new model requires a new image; fetch weights at startup from storage or a registry instead. Datasets copied in by an unfiltered copy instruction. Package caches left behind in the layer that created them. The repository's version-control directory, notebooks, checkpoints and logs, swept in because the build context was the whole directory. Write the ignore file before the first build, listing data, notebooks, caches, the version-control directory and anything large — it is thirty seconds and it prevents an image nobody can explain the size of.

Accelerator containers are where the confusing failures live. A container using an accelerator needs the toolkit inside the image to be compatible with the driver on the host, and the container runtime has to expose the device at all. When they do not match, the failure is not a clear message — it is a library that cannot initialise, or a framework reporting no accelerator available on a machine that plainly has one, or an obscure error deep in a numerical library.

Knowing that this compatibility question exists is most of the value, because it converts a long confusing afternoon into a specific check. The practical rules: start from an official base image that already carries a matching toolkit rather than assembling one; pin the base image version rather than following a moving tag; record the driver version of the host you deploy to and confirm the pairing before you need it; and remember the image is not portable across hosts with incompatible drivers, however portable containers are in general.

A compose file is what makes local development resemble production. A real system is rarely one process — a service, a database, an object store, a cache, a tracking server. Describing them together means the whole stack starts with one command, on a new machine, with the versions everyone else is using. It is also the fastest way for somebody new to run your project, and the honest test of whether your setup instructions are complete: if the compose file starts it, they are.

Two habits worth carrying. Pin versions — the base image, the dependencies, the toolkit — because an unpinned build is reproducible only until an upstream release, and then it is a different environment with your commit hash on it, which is exactly the reproducibility the versioning topic was trying to protect. And run as a non-root user inside the container, one line, which limits what a compromise can do and is expected by most production platforms.

What you should now be able to explain or do

Explain layer caching and its downstream effect, and order instructions so rebuilds are fast. Write a multi-stage build and say what the final stage contains. Name the four sources of bloat in this field and write the ignore file first. Fetch model weights at startup rather than baking them in, and say why. State the accelerator compatibility question and the four practical rules. Use a compose file to start a local stack and treat it as the test of your setup instructions. Pin versions and run as a non-root user.

Check yourself