5.7 Metrics and calibration
You can pick the metric that matches the cost of being wrong.
Before:03. Data Handling & AnalysisUnlocks:06. Deep Learning10. Production & MLOps13. Capstones, Portfolio & Interviews
Metric choice encodes the cost of being wrong: precision against recall, MAE against RMSE — each pair disagrees exactly when the decision matters. Calibration asks the further question of whether predicted probabilities mean what they say. It follows cross-validation because a trustworthy number needs both an honest split and a fitting metric. The canonical warning is accuracy on imbalanced data: ninety-nine percent accuracy on one-percent positives describes a model that may have learned nothing at all.
Work through these
Accuracy, precision, recall, F1, and their trade-offs
The standard classification measures, each answering a different question about a different kind of mistake. Reporting only overall accuracy hides everything that matters on unbalanced problems.
ROC-AUC vs. PR-AUC on imbalanced data
Two summary curves that behave very differently when one class is rare, and the second is the more informative in that case. Choosing between them is not a matter of taste.
Regression metrics: MAE, RMSE, MAPE, R²
The regression measures differ in how they treat large errors and whether they are scale-free. Picking one should follow from what a large error actually costs.
Probability calibration: Platt, isotonic, reliability curves
A model can rank well and still output probabilities that are systematically wrong, which matters whenever the number itself is used. Two correction methods and one diagnostic plot are covered here.
Sign in to keep your progress.
Free resources
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.