2.14 Modules, packaging and environments

What this is and why it exists

Packaging is what turns "works on my machine" into "install it and run it", and for a data or machine-learning project it is also what turns a result into a reproducible one. This topic covers how imports actually resolve — which explains most mysterious import errors — how a project declares itself, and why a lockfile is the difference between a promise of reproducibility and the fact of one.

The vocabulary

• Module — one Python file.
• Package — a directory of modules that can be imported as a unit.
• Absolute import — naming the module from the top of the package.
• Relative import — naming it relative to the current module, with leading dots.
• Project file — pyproject.toml, the single declaration of a project's metadata, dependencies and build settings.
• Virtual environment — an isolated set of installed packages belonging to one project.
• Dependency specification — what you asked for, as a range.
• Lockfile — what you actually got, pinned exactly, so the next install matches.

The mental model

An import searches a list of locations, in order, and the first match wins. That single sentence explains almost every confusing import error you will meet. A file of your own named after a library shadows the library, because your directory is searched first — a random.py in your project makes the standard library's version unreachable, and the error appears somewhere that never mentions your file. Running a script directly and importing it as part of a package give different starting points for that search, which is why relative imports work in one and fail in the other. When an import misbehaves, ask what the search path is and where the name was found; guessing at syntax rarely helps.

Use absolute imports within a package by default: they say where the thing is, they survive a file being moved, and they read the same wherever they appear. Relative imports have a legitimate place inside a package that may be renamed, and they are the reason a package imported as a script behaves oddly.

The project file is the modern single source of declaration. It holds the name, the version, the dependencies, the build backend and the tool configurations that used to be scattered across several files. Having one file means there is one place to look, which sounds administrative and turns out to matter every time somebody new opens the repository.

Then the distinction this topic exists for. A dependency specification is a wish; a lockfile is a fact. A specification saying "a version of this library at or above 2.1" is what you want, and it resolves differently in March and in June because new releases exist. A lockfile records the exact versions that were installed, including the dependencies of your dependencies, so the same repository installs identically today and in six months. Without one, a training run you cannot reproduce is not a mystery — it is the expected outcome, because the code was the same and the libraries were not. Commit the lockfile, and treat updating it as a deliberate change with its own commit.

Modern tools — uv, Poetry, pip-tools — all provide the same two-file arrangement over the same project file: you edit the specification, the tool resolves and writes the lock, and installs come from the lock. Any of them is fine and the choice matters far less than actually having a lockfile.

The virtual environment is the isolation that makes all of it work: each project gets its own installed packages, so two projects can want different versions of the same library without a fight, and installing something for one cannot break another. Never install a project's dependencies into the system Python — the day two projects disagree, you have no way out.

And the final item is worth doing once for its own sake: publish something small. Building a distribution and putting it on the package index is an afternoon, and it converts packaging from a topic you have read about into a thing you have done — after which the version numbers, the metadata and the build step stop being abstract.

In code

Checked against the Python packaging guide, which describes pyproject.toml as "a configuration file used by packaging tools, as well as other tools such as linters, type checkers, etc.", the build-system table as declaring "which build backend you use and which other dependencies are needed to build your project", and requires-python as letting you "declare the minimum version of Python that you support".

A minimal project file:

[project]
name = "scoretools"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = [
    "pandas>=2.2",
    "pydantic>=2.7",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

And the layout that avoids the commonest import surprise:

scoretools/
  pyproject.toml
  uv.lock
  src/
    scoretools/
      __init__.py
      io.py
      scoring.py
  tests/
    test_scoring.py

Putting the package under src/ means your tests import the installed package rather than the directory they happen to be sitting next to — so a test passing locally and failing after install becomes impossible, because there is no version of the code that only works from the project root.

What you should now be able to explain or do

Explain how an import resolves and use that to diagnose a shadowed module. Say when relative imports are appropriate and why they break in a script run directly. Write a minimal project file with a name, a version and dependencies. State the difference between a specification and a lockfile in one sentence each, and say what is lost without the second. Explain what a virtual environment isolates. Lay a project out so that tests exercise the installed package.

Check yourself