2.15 Testing with pytest

What this is and why it exists

Tests are not about proving code correct; they are about being able to change it. Six months from now you will want to restructure a pipeline, and the only thing that makes that a Tuesday afternoon rather than a fortnight of anxiety is a suite that fails loudly when you break something. pytest makes writing them cheap enough that there is no excuse — plain functions, plain assertions, and fixtures for anything that needs setting up.

The vocabulary

• Test function — an ordinary function whose name begins with test_, which pytest discovers and runs.
• Assertion — a plain assert, which pytest rewrites so that a failure shows the actual values.
• Fixture — a reusable piece of setup a test asks for by naming it as a parameter.
• Scope — how long a fixture lives; the default is per test function.
• Parametrise — running the same test over several inputs, each reported separately.
• Monkeypatch — replacing something temporarily for the duration of a test.
• Mock — a stand-in object recording how it was called.
• Coverage — the proportion of lines the suite executed.

The mental model

pytest finds functions whose names begin with test_ and runs them; a test passes if it does not raise. That is the whole framework at the level you need it, and it is why the barrier to writing the first test is a single file with one function in it.

Fixtures are the second idea and the reason the framework scales. The documentation describes the mechanism plainly: "test functions request fixtures they require by declaring them as arguments. When pytest goes to run a test, it looks at the parameters in that test function's signature, and then searches for fixtures that have the same names as those parameters." So a test that needs a temporary directory, a sample dataframe or a configured object asks for it by name and receives it — and the setup lives in one place instead of at the top of thirty tests.

Scope decides how often that setup runs. The default is per function, which the documentation describes as the fixture being "destroyed at the end of the test"; module and session scopes reuse the same object across a file or a whole run. Wider scope is faster and riskier: anything a test mutates is then visible to the next one, and tests that pass alone and fail together are among the most tiresome bugs there are. Default to function scope and widen only for something genuinely expensive and genuinely read-only.

Parametrising is the cheapest coverage you will ever buy. One test body plus a list of inputs becomes many reported cases, so the empty input, the single item, the duplicate, the negative number and the enormous value are five lines rather than five tests — and each failure names the input that caused it.

Mocking and monkeypatching are for cutting the test off from the world: the interface call, the clock, the file that may not exist, the model that takes four minutes. Use them at the boundary and sparingly. A test suite that mocks its own internals is testing that your code calls the functions you wrote, which is a tautology and will keep passing while the behaviour is wrong.

Then the honest word about coverage, which is this topic's trap. Coverage measures which lines ran, not whether the behaviour is right. A suite that calls every function and asserts nothing reports full coverage and proves nothing at all; meanwhile a single thoughtful test of the boundary case where your rounding is wrong is worth more than a hundred lines of exercise. Use coverage the way it is genuinely useful — as a map of what nobody has looked at — and never as a target, because a number that becomes a target gets met the cheap way.

For a data project, one specific habit pays for itself repeatedly: test the transformations rather than the model's accuracy. Whether the model reaches a given accuracy is a research result and it fluctuates; whether your feature function drops rows with a missing value, whether your split leaks the same person into both sides, whether your scaler was fitted on the training set only — those are facts with right answers, they break silently, and they are exactly what a test catches.

In code

Checked against the pytest fixtures documentation.

import json
import pytest

from scoretools.scoring import normalise, average


@pytest.fixture
def rows():
    """Fresh sample data per test, so no test can affect another."""
    return [
        {"name": "asha", "score": 88},
        {"name": "ravi", "score": 71},
    ]


def test_average_of_two(rows):
    assert average(rows) == 79.5


@pytest.mark.parametrize(
    "raw,expected",
    [(0, 0.0), (50, 0.5), (100, 1.0)],
)
def test_normalise_scales_to_unit_range(raw, expected):
    assert normalise(raw) == expected


@pytest.mark.parametrize("bad", [-1, 101, None, "eighty"])
def test_normalise_rejects_impossible_values(bad):
    with pytest.raises(ValueError):
        normalise(bad)


def test_reads_from_a_file(tmp_path, rows):
    """tmp_path is a built-in fixture: a fresh directory, cleaned up after."""
    path = tmp_path / "scores.json"
    path.write_text(json.dumps(rows), encoding="utf-8")
    assert average(json.loads(path.read_text(encoding="utf-8"))) == 79.5

Notice that the four bad values are one test, not four, and that each will be reported separately with the value that failed. Notice too that the fixture returns fresh data every time it is requested — had it returned a module-level list, one test appending to it would change what another test sees, and the failure would depend on the order they happened to run in.

What you should now be able to explain or do

Write a test file pytest discovers, with no configuration. Explain how a test asks for a fixture and what pytest does with the parameter name. Choose a fixture scope and say what widening it risks. Parametrise a test over five inputs including the awkward ones. Say where mocking belongs and what a suite that mocks its own internals is really testing. State what coverage proves and what it does not. Name three things worth testing in a data pipeline that are not the model's accuracy.

Check yourself