5.6 Cross-validation and honest evaluation

What this is and why it exists

A score is a promise about data you have not seen. Cross-validation is how that promise is estimated, and the estimate is only honest if the split mirrors the way the model will actually be used. Get the split wrong and information leaks across it: the number looks excellent, everybody believes it, and the failure surfaces in production where it costs the most.

The vocabulary

• Training set — what the model learns from.
• Validation set — what you use while choosing between models and settings.
• Test set — looked at once, at the end, to report a number.
• k-fold — splitting into k parts and using each in turn as the held-out part.
• Stratified — keeping the class proportions the same in every fold.
• Grouped — keeping all rows belonging to one subject on the same side.
• Time-series split — training on the past and testing on the future, always.
• Nested cross-validation — an inner loop for tuning inside an outer loop for estimating.

The mental model

Three sets, three jobs. The training set fits the model. The validation set chooses between models — every comparison you make, every setting you tune, every threshold you pick. The test set exists to answer one question once: how well does the chosen thing do? Look at it twice and it stops being a test set, because from the second look onwards you are choosing against it.

The mechanics of k-fold are straightforward — the reference describes it as dividing the samples into k groups, with the prediction function learned from all but one of them while "the fold left out is used for test" — and the interesting part is which variant. The split strategy must reproduce the gap between training and deployment. Ask what the model will be asked about that it has not seen, and split along that.

Stratified keeps the class balance even, which the reference describes as folds where "each set contains approximately the same percentage of samples of each target class as the complete set". With a rare positive class this stops a fold containing almost none of it, which would otherwise make the score meaningless and unstable. Use it by default for classification.

Grouped keeps a subject whole. The documentation gives the case exactly: "if the data is obtained from different subjects with several samples per-subject and if the model is flexible enough to learn from highly person specific features it could fail to generalize to new subjects". Several rows per student, per patient, per device — if the same subject appears on both sides, the model can recognise the subject rather than learn the pattern, and the score measures memory.

Time-series keeps the arrow of time. The strategy returns the first k folds as training and the next as test, so every evaluation trains on the past and predicts the future — which is what the model will have to do. A random split on temporal data trains on Thursday to predict Tuesday, and the resulting backtest is, in the module's own phrase, a pleasant lie.

Then tuning, and the subtlety that nested cross-validation exists for. If you use the validation folds to choose settings and to report the score, the score is optimistic: you selected the settings that happened to suit those folds. Nested cross-validation separates the jobs — an inner loop tunes within each outer training portion, and the outer loop scores the whole procedure on data untouched by the tuning. It costs k times more compute and it is what you use when the number has to be defensible rather than encouraging.

Finally, the failure mode this topic ends on, because it is the most common one among people who are otherwise careful. Leaderboard overfitting. Evaluate two hundred variants against the same validation set, pick the best, and its advantage is partly real and partly luck — you have selected for whichever variant happened to suit that particular sample. It is the same mistake as looking at the test set twice, stretched over a fortnight and made to feel like diligence. The defences: keep a final test set genuinely untouched until the end; be suspicious of an improvement smaller than the fold-to-fold spread; and count how many comparisons you have made, because the more you make, the more of your best score is noise.

In code

Checked against the scikit-learn user guide.

from sklearn.model_selection import (
    StratifiedKFold, GroupKFold, TimeSeriesSplit, cross_val_score,
)

# Classification with an uneven class balance.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

# Several rows per student: keep each student wholly on one side.
cv = GroupKFold(n_splits=5)
scores = cross_val_score(pipeline, X, y, cv=cv, groups=student_ids, scoring="average_precision")

# Anything with a time axis: train on the past, test on the future.
cv = TimeSeriesSplit(n_splits=5)

print(scores.mean(), scores.std())   # report both; the spread is information

The last line is the habit worth keeping. A mean without a spread hides whether the estimate is stable, and the spread is exactly what tells you whether a rival model's small advantage means anything.

What you should now be able to explain or do

Say what each of the three sets is for and what looking at the test set twice costs. Choose a split strategy from a description of the data and defend it. Say what a random split does to grouped data and to temporal data, and why the resulting score is worse than useless. Explain what nested cross-validation separates and when it is worth its cost. Report a mean with a spread and use the spread to judge whether an improvement is real. Recognise leaderboard overfitting in your own work.

Check yourself