10.1 From notebook to package

What this is and why it exists

A notebook is an excellent place to think and a poor place to keep anything. This topic is the jump to software: a layout somebody else can find their way around, settings in files rather than edited into cells, one entry point for the common commands, and training, prediction and evaluation kept apart. It is the least glamorous topic in the module and it is the one that decides whether your work can be run by anybody but you.

The vocabulary

• Package — importable code with a defined structure, rather than a script.
• Configuration — settings held outside the code, describing a run.
• Entry point — a named command that does one thing.
• Task runner — a file collecting those commands so nobody has to remember them.
• Data boundary — the line between what lives in the repository and what does not.
• Execution-order state — the invisible dependence of a notebook on the order cells were run.
• Artefact — a file a run produced: a model, a report, a set of predictions.

The mental model

The specific problem with a notebook is invisible state. Cells can be run in any order, and the variables in memory depend on the order you happened to use — including cells you edited afterwards and cells you deleted. A notebook that produces the right answer on your machine can produce a different one when run top to bottom, and there is no way to tell by looking. The number in the output was produced by a program that no longer exists, and that is where reproducibility dies.

So the notebook keeps its real job: exploring, plotting, trying an idea in ten minutes. What it stops being is where anything lives. The moment a piece of code has been useful twice, it moves into the package and the notebook imports it.

A conventional layout is worth adopting exactly because it is conventional. Source under one directory, tests beside it, configuration in its own, notebooks in theirs, scripts as thin entry points, and data outside the repository. The value is not any particular arrangement — it is that a person who has seen this layout before can find their way around yours without asking, and can guess where a new file belongs.

The data boundary is the part people get wrong. Data does not go in the repository: source control handles large binary files badly, a dataset copied into a repository is there permanently even after deletion, and personal data committed once is a disclosure you cannot undo. Keep data in object storage or on a shared volume, keep a pointer in the repository — a path, an identifier, a checksum — and use the versioning tools from the data-versioning topic to make that pointer meaningful. Add the data directories to the ignore file on the first commit, before anything can slip in.

Configuration replaces the magic numbers. A learning rate written into a cell, a file path typed three times, a threshold somebody chose on a Tuesday: those are what make a run impossible to describe. Put them in a configuration file, load them at the start, and print the whole resolved configuration into the log. Then a run can be described rather than remembered: a file plus a commit is the complete account of it, which is exactly what the experiment-tracking topic then records.

Three rules make configuration work. No setting exists in two places — a value in a file and a default in the code that disagree is a bug waiting for the day somebody reads the wrong one. Secrets go in the environment, never in the configuration file, because configuration is committed and secrets must not be. And compose rather than duplicate: a base configuration with small overrides per experiment beats twelve near-identical files, and it is what the dedicated configuration libraries exist to provide, along with the ability to change a setting from the command line without editing anything.

A task runner is a small file that removes a surprising amount of friction. One place where train, evaluate, serve, test, lint and format are defined, so nobody reconstructs a long invocation from memory or from a colleague's shell history. It documents the project's commands by existing, it is what a new person runs first, and it is what the automated pipeline calls, so the commands that run in the pipeline are the commands you run locally — which removes an entire class of "it passes on my machine".

Separating training, inference and evaluation is the last piece and the one with the longest payoff. Left together they drift into one script with flags that does all three badly, and then the preprocessing used at prediction time is a slightly different path from the one used in training — which is the training-serving skew the pipelines topic is about, arriving through the code layout.

Keep three modules with three responsibilities. Training produces a model artefact and nothing else. Inference loads an artefact and produces predictions, with no training code imported at all. Evaluation takes predictions and truth and produces numbers, and knows about neither model nor training.

Two benefits follow immediately. Each is independently testable: evaluation can be tested with invented predictions, inference with a stub model, training with two rows of data. And the shared preprocessing has to be extracted into one place both training and inference import — which is the pipeline object from the classical module, and is the mechanical reason the two cannot drift.

The trap this topic exists for is "I'll structure it later". Later does not come, because by then the notebook has produced a result somebody is waiting for and restructuring risks changing the number. Start with the layout on the first day, when it costs twenty minutes and there is nothing to break. The cost of adopting it at the start is small enough that it is not worth deciding; the cost of adopting it after six months is a project.

What you should now be able to explain or do

Say exactly why notebooks lose reproducibility, and what a notebook is still for. Lay out a repository conventionally and say why convention itself is the value. Keep data out of the repository, with a pointer in it and ignore rules from the first commit. Move settings into configuration, print the resolved configuration, and apply the three rules. Write a task runner and use the same commands locally and in the pipeline. Separate training, inference and evaluation, and say what each may import. Explain why the shared preprocessing must be extracted.

Check yourself