3.5 Data cleaning and leakage

What this is and why it exists

Cleaning decides whether a model result means anything. Every choice here — what to do about missing values, which outliers to keep, how to deduplicate — changes the conclusion, and one of them, leakage, is capable of producing a validation score that looks excellent and is entirely fictional. This is the most expensive mistake in applied machine learning, it takes almost nothing to do by accident, and the whole topic is arranged around preventing it.

The vocabulary

• Missing completely at random — whether a value is missing has nothing to do with anything.
• Missing at random — whether it is missing depends on other observed columns.
• Missing not at random — whether it is missing depends on the missing value itself.
• Imputation — filling a missing value with an estimate.
• Outlier — a value far from the rest, which may be an error or the most important record you have.
• Deduplication — removing records that represent the same thing twice.
• Leakage — information reaching a feature that would not be available at prediction time.
• Contamination — test data influencing anything about the training process.

The mental model

Start with missingness, because the reason a value is absent decides what you may do about it. If missingness is unrelated to everything, dropping those rows loses data and biases nothing. If it depends on other columns you have — younger students skipping a question more often — you can model it, and a sensible imputation is honest. If it depends on the missing value itself — people with the lowest scores declining to report them — then no imputation from the observed data can recover it, and filling the gap with a mean actively invents a pattern that is not there. You usually cannot prove which case you are in, so the honest procedure is to ask what would cause a value to be missing here, write the answer down, and choose accordingly.

Two practical rules make this safer regardless. Add a "was missing" indicator column when you impute, so the model can learn that absence itself was informative. And never let the fact of imputation disappear from the record, because a downstream reader who does not know a column was filled will draw conclusions from your estimates as though they were measurements.

Outliers get a three-step treatment and the middle step is the one people skip. Detect, understand, then decide. A student with a score of 900 is a data-entry error. A transaction a thousand times larger than the rest may be the fraud you are trying to find. Deleting outliers because they are inconvenient is how a model becomes blind to exactly the cases it was built for — so the question is never "is this far from the others" but "what does this record actually represent".

The unglamorous problems are duplicates, inconsistent categories and units, and they cause more wrong answers than anything sophisticated. The same student twice under two identifiers inflates every count. A subject column holding both "maths" and "Maths" splits a group in half. A column mixing metric and imperial produces a distribution with two humps that somebody will interpret as two populations. All three are found by looking: distinct value counts, exact and near-duplicate checks, and a histogram of every numeric column.

Then leakage, which deserves the rest of this lesson. A feature must contain only what would be known at the moment of prediction. Leakage arrives in three shapes. Information from the future: an aggregate computed over the whole dataset, a status field updated after the outcome, a rolling window that is centred. Information from the target: a column derived from what you are predicting, of which the classic is a target-mean encoding computed on all the data. And contamination: the test set influencing the training process — fitting a scaler or an imputer on everything before splitting, choosing features by looking at the whole dataset, or tuning against the test set until it is no longer a test.

The defence is one rule with a shape you can check: split first, then fit everything on the training portion only, and apply to the rest. The scaler's mean, the imputer's median, the encoder's categories, the feature selection — all learned from training data, all applied unchanged to validation and test. A pipeline object enforces this by construction, which is why the next module builds everything as pipelines.

And the tell to remember, because you will meet it: a model that performs implausibly well is not a triumph, it is a symptom. When the validation score is far better than the problem should allow, look for the column that knows the answer.

What you should now be able to explain or do

Name the three missingness mechanisms and say which one no imputation can fix. Add an indicator column and say what it lets the model learn. Apply the three-step treatment to an outlier and say why deleting first is dangerous. Find duplicates, inconsistent categories and unit errors by looking rather than by hoping. Name the three shapes of leakage with an example of each. State the split-then-fit rule and say what enforces it. Say what an implausibly good score means.

Check yourself