5.7 Metrics and calibration

What this is and why it exists

A metric is a statement about what it costs to be wrong, and choosing one is therefore a business decision wearing a technical hat. Different metrics disagree, and they disagree precisely in the cases that matter. This topic also contains the most misleading number in applied machine learning — accuracy on imbalanced data — and the check that tells you whether your probabilities mean anything at all.

The vocabulary

• True positive, false positive, false negative — caught it, cried wolf, missed it.
• Precision — of the cases you flagged, the share that really were positive.
• Recall — of the real positives, the share you caught.
• F1 — the harmonic mean of the two, which punishes a large imbalance between them.
• ROC-AUC — how well the model ranks positives above negatives across all thresholds.
• PR-AUC (average precision) — the same idea using precision against recall.
• MAE — mean absolute error; RMSE — root mean squared error.
• Calibration — whether a predicted probability matches the observed frequency.

The mental model

Precision and recall are the two questions you can ask about a flag, and they pull in opposite directions. Lower the threshold and you catch more real positives (recall rises) while flagging more things wrongly (precision falls). Raise it and the reverse. Neither is better; which one you want is decided by the cost of each mistake, and the honest way to state a requirement is "we need to catch at least 90 percent, and we can tolerate looking at three false alarms per real case", which pins both.

F1 combines them and is a reasonable default when you have no cost information, with one caveat: it treats precision and recall as equally important, which is exactly the assumption you were trying to avoid making by accident.

Then accuracy, and the reason this topic exists. On a dataset with one percent positives, a model that always says "no" is 99 percent accurate and has learned nothing whatsoever. Accuracy is dominated by the majority class, so on imbalanced data it measures the imbalance rather than the model. The alternatives that do not have this problem are precision, recall, and the area under the precision-recall curve.

ROC-AUC and PR-AUC both summarise a model across all thresholds, and the difference between them is worth knowing because it is a common mistake. ROC-AUC plots the true-positive rate against the false-positive rate, and the false-positive rate has the number of negatives in its denominator — so with a hundred thousand negatives, several hundred false alarms barely move it. A model can therefore have a fine ROC-AUC while being useless in practice. PR-AUC uses precision, whose denominator is what you flagged, so those false alarms show up immediately. On imbalanced problems, report PR-AUC. ROC-AUC remains a reasonable summary when the classes are roughly balanced.

For regression, the choice between MAE and RMSE is a choice about large errors. RMSE squares them before averaging, so it is dominated by the worst cases and is right when a large error is disproportionately bad. MAE treats every unit of error alike and is right when it is not, and it is also more robust to a few extreme values. MAPE expresses error as a percentage, which is readable and breaks near zero — where dividing by a tiny actual value makes the percentage enormous — so it is unsuitable for anything that can be near zero. And R-squared reports the proportion of variance explained, which is useful for comparing models on the same data and misleading across datasets, because it depends on how much variance there was to explain.

Then calibration, which is a separate question from accuracy and is often the one that matters. The documentation defines it exactly: a well-calibrated classifier "should classify the samples such that among the samples to which it gave a predict_proba value close to, say, 0.8, approximately 80% actually belong to the positive class". You check it with a reliability diagram, which compares "the average predicted probability in each bin" with "the fraction of positives" in that bin — a perfectly calibrated model traces the diagonal, a model above it is under-confident and one below is over-confident.

And you fix it by fitting a small model on top. Calibration means "fitting a regressor (called a calibrator) that maps the output of the classifier… to a calibrated probability", and there are two standard choices. The sigmoid method "assumes the calibration curve can be corrected by applying a sigmoid function to the raw predictions" and suits small samples. The isotonic method fits a step-wise non-decreasing function and "is more general… as the only restriction is that the mapping function is monotonically increasing… However, it is more prone to overfitting, especially on small datasets" — the documentation's rule of thumb being that isotonic matches or beats sigmoid once there is enough data, above roughly a thousand samples. Crucially the calibrator must be fitted on data the classifier did not train on, which is why the tool that does this uses cross-validation internally.

What you should now be able to explain or do

State precision and recall in one sentence each and say which way each moves with the threshold. Write a requirement that pins both. Explain why accuracy is misleading on imbalanced data, with the 99-percent example. Say why ROC-AUC can look fine while a model is useless, and what to report instead. Choose between MAE and RMSE from what a large error costs, and say when MAPE breaks. Define calibration, read a reliability diagram, and choose between the two calibration methods by sample size.

Check yourself