10.9 Data pipelines and feature stores

What this is and why it exists

Training-serving skew is a disease with one cause: the same feature computed by two different pieces of code. They agree on the day they are written and they diverge afterwards, silently, and the model underperforms for reasons nobody can find because both paths look correct. This topic is the orchestration that makes multi-step data flows reliable, the quality checks that stop bad data at the door, and the one architectural answer that removes the duplication rather than managing it.

The vocabulary

• Pipeline — a sequence of data steps with dependencies between them.
• Orchestrator — the system that runs those steps on a schedule and in order.
• Backfill — re-running a pipeline over past periods.
• Idempotent step — one that can be re-run for the same period without duplicating.
• Batch ingestion — data arriving on a schedule; streaming — continuously.
• Feature store — one place features are computed and served from.
• Point-in-time correctness — using only what was known at the moment being modelled.
• Quality gate — a check that stops a pipeline when the data looks wrong.

The mental model

An orchestrator is what makes a collection of scheduled scripts into a system. Scripts scheduled independently have no notion of dependency, so a downstream job runs whether or not its input arrived, and a failure at three in the morning is discovered by its consequences. An orchestrator holds the dependencies, runs steps in order, retries on failure, records what succeeded, allows a re-run of one step or of a past period, and shows the whole flow.

Three are widely used and differ in emphasis — one is the long-established default with the largest ecosystem, one is lighter and closer to ordinary Python, one organises around the data assets a pipeline produces rather than the tasks that produce them. The choice matters far less than having one at all, and the third framing is worth knowing regardless: thinking in terms of "this dataset is stale" rather than "that job failed" is closer to the question you actually care about.

Two properties matter more than the tool. Steps must be idempotent per period: re-running yesterday's step must produce the same result rather than duplicating rows, because you will re-run it — after a failure, after a fix, after an upstream correction. Write the period's output as a replacement rather than an append, and backfills become ordinary instead of dangerous. And steps should be small, because a step is the unit of retry and of visibility, and one script that does everything gives you no information about where it stopped.

Batch and streaming are both needed, for different features. Batch is simple, cheap, testable and straightforward to reason about, and it is correct whenever the feature can be minutes or hours old — which covers most features in most systems. Streaming is required when freshness is part of the answer: the count of actions in the last minute, the current session's behaviour, a fraud signal that is worthless the moment it is late.

The rule is to use batch until a feature's staleness demonstrably costs you something, because streaming brings out-of-order events, late arrivals, exactly-once concerns and a much harder debugging story. Many systems end with both, and the danger of both is the next section: the same feature defined twice, once in each.

Now the disease. A feature is defined in the training pipeline, in Python, over historical data. The same feature is defined again in the serving path, in whatever language and framework the service uses, over live data. They agree when written. Then one is fixed — a null handled, a boundary adjusted, a unit corrected, a time zone — and the other is not, because the person changing it did not know the other existed.

The model now sees, at prediction time, values that differ from anything it was trained on, and nothing errors. The service is healthy, the pipeline is green, the accuracy is quietly worse. This is the single most common reason a model that evaluated well performs poorly in production, and it is entirely structural: two implementations of one definition will diverge, and no amount of care prevents it.

A feature store exists to remove the duplication rather than to manage it. A feature is defined once, in one place, and the store computes it and serves it to both sides: an offline interface producing historical values for training, and an online interface producing current values for serving, from the same definition. The duplication is gone, so it cannot diverge.

The subtler thing a feature store provides is point-in-time correctness, and it is worth understanding because it is a leakage problem in new clothes. To build a training row for an event last March, you need each feature's value as it was in March, not as it is now. A naive join against a table of current values gives the model information from after the event — the classic leakage, producing an excellent evaluation and a disappointing deployment. The store keeps values with their valid times and joins as of the event's timestamp, which is fiddly to get right by hand and is most of what these systems are for.

And the honest note: a feature store is real infrastructure, and it is not always warranted. For a handful of features and one model, the cheaper answer is the same one the classical module gave — one shared library, imported by both the training pipeline and the serving code, with the feature definitions in it and tests asserting that both paths produce identical output for the same input. That test in the release pipeline catches divergence at the commit that causes it. Adopt the store when you have many features, several models sharing them, or a genuine need for the point-in-time joins.

Quality gates are the last piece, and the argument for them is arithmetic. A bad value caught at ingestion costs one alert. The same value reaching a model costs a retrain, and reaching production costs an incident and every decision made in between. Failing loudly at the boundary is the cheapest place to fail.

What to check, at every ingestion: the schema contract from the versioning topic — columns, types, nullability, ranges, permitted categories; volume, since a file with a tenth of the usual rows is a broken upstream job and not a quiet week; freshness, since data that did not arrive is a failure that a pipeline reading yesterday's file will not notice; and distribution, comparing against recent history to catch a unit change or a shifted encoding that passes every type check.

Stop the pipeline on a failure rather than warning. A warning in a log is a decision to continue with bad data, made by nobody. And route the failure to a person with the specific check, the expected value and the actual one, because "quality check failed" costs an hour that "the amount column has 12% nulls, expected under 1%" does not.

What you should now be able to explain or do

Say what an orchestrator provides over scheduled scripts, and why per-period idempotency and small steps matter more than the tool. Choose batch over streaming until staleness demonstrably costs something. Explain training-serving skew as a structural consequence of two implementations. Say what a feature store removes and what point-in-time correctness prevents. Choose the shared-library alternative and test both paths for identical output. Implement quality gates on schema, volume, freshness and distribution, failing the pipeline with a specific message.

Check yourself