5.6 Cross-validation and honest evaluation
Your reported score survives contact with new data.
Before:03. Data Handling & AnalysisUnlocks:06. Deep Learning10. Production & MLOps13. Capstones, Portfolio & Interviews
Cross-validation estimates performance on unseen data, and the split must mirror how the model will be used: stratified for imbalance, grouped for related rows, time-ordered for anything temporal. It sits at the heart of the module because every later score is only as honest as its split. The leakage to fear is structural — random splits on time series or grouped data let information cross the fence, and the score that results is a promise production will not keep.
Work through these
Train/validation/test discipline
Three separate sets with three separate purposes, and one rule that matters more than the rest: the final set is touched once, at the end. Every violation of that rule inflates the reported score.
k-fold, stratified, grouped, time-series splits
Different splitting schemes for different data: preserving class proportions, keeping related records together, and respecting time order. Using the wrong one leaks information and produces a score that will not hold.
Nested CV for tuning plus evaluation
When tuning and evaluation both use the same data, the reported score is optimistic. Nesting one procedure inside the other is the correct fix, and it costs more computation.
The leaderboard-overfitting failure mode
Repeatedly checking against the same held-out set gradually fits the model to it, which is why competition leaderboards flatter their leaders. Knowing this is what keeps your own numbers honest.
Sign in to keep your progress.
Free resources
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.