5.19 scikit-learn pipelines end to end

What this is and why it exists

A pipeline binds every preprocessing step and the model into one object, so that whatever transformations training saw, prediction sees identically. It is the single habit in this whole module that prevents leakage and production skew at the same time, and it costs about four extra lines. If you take one practice from Classical Machine Learning into everything you build afterwards, take this one.

The vocabulary

• Transformer — an object with a fit step and a transform step: learns something from data, then applies it.
• Estimator — an object that fits and predicts.
• fit / transform / predict — the shared interface everything in the library follows.
• Pipeline — a sequence of transformers ending in an estimator, behaving as one estimator.
• ColumnTransformer — different treatment for different columns, combined into one step.
• Custom transformer — your own step, written to the same interface so it composes.
• Persistence — saving the fitted object so that inference uses exactly what training produced.

The mental model

Everything follows one interface. A transformer's fit learns from data — a scaler learns means and spreads, an imputer learns medians, an encoder learns which categories exist — and its transform applies what it learned. An estimator's fit trains and its predict predicts. Because everything shares that shape, they compose.

A pipeline chains them and is itself an estimator. Fitting it fits each step in turn on the output of the previous one; predicting runs each transform and then the model. The documentation lists what that buys, and the third item is the one that matters most here: convenience and encapsulation, so "you only have to call fit and predict once"; joint parameter selection, so "you can grid search over parameters of all estimators in the pipeline at once"; and safety — pipelines "help avoid leaking statistics from your test data into the trained model in cross-validation, by ensuring that the same samples are used to train the transformers and predictors."

Read that last sentence next to the cleaning topic and the whole module joins up. Fit a scaler on the full dataset and split afterwards, and the scaler's mean came partly from validation rows — quiet leakage, an optimistic score, and no error message anywhere. Put the scaler in a pipeline and pass the pipeline to cross-validation, and the scaler is refitted inside every fold on that fold's training portion only. The rule stops being something you have to remember correctly under time pressure and becomes a property of how the code is arranged.

The same object solves production skew, which is the other half. Whatever the pipeline learned at training — the medians, the category lists, the scaling constants — travels with it when you save it. There is no second implementation of the preprocessing to drift out of step with the first, because there is only one, and it is the fitted one.

ColumnTransformer is what makes this usable on real tabular data, where columns need different treatment: scale the numbers, impute and one-hot the categories, pass a few through untouched. The documentation describes it as performing "different transformations for different columns of the data, within a Pipeline that is safe from data leakage and that can be parametrized" — and names the problem it exists for, that "incorporating statistics from test data into the preprocessors makes cross-validation scores unreliable".

Custom transformers are how domain knowledge joins the same protection. Write a class with a fit that learns whatever it needs and a transform that applies it, and it composes with everything else and is refitted per fold like the rest. The design rule is the one from the cleaning topic: anything learned from data belongs in `fit`, and only application belongs in `transform`. A transform that computes a mean from the data it is given at transform time has reintroduced exactly the leakage the pipeline was preventing.

Persistence is the last step and it has two requirements people discover the hard way. Save the fitted pipeline, not the model alone — a model without its preprocessing is not a thing that can predict. And pin the versions of the libraries that produced it, because a saved object is not a stable format across library versions: it may fail to load, or worse, load and behave differently. Record the library versions, the training data's identity, and the code commit alongside the file. The pipeline makes the transformations reproducible; those three facts make the object reproducible.

What you should now be able to explain or do

Describe the shared interface and say why it lets things compose. Say what a pipeline does when it is fitted and when it predicts. Quote the safety property in your own words and connect it to the leakage rule. Explain how the same object also prevents production skew. Use a column transformer to treat numeric and categorical columns differently. Write a custom transformer that keeps the rule about what belongs in fit. Say what to save and what to record beside it.

Check yourself