5.17 Imbalanced data

What this is and why it exists

The problems worth solving are usually imbalanced. Fraud, defects, churn, rare diseases — the interesting class is a small minority, and every default in every library assumes it is not. This topic collects the honest responses, in the order they should be tried, and names the one shortcut that produces impressive validation scores which are entirely fake.

The vocabulary

• Class imbalance — one class far outnumbering the other.
• Majority-class baseline — always predicting the common answer, which is often surprisingly accurate.
• Undersampling — discarding majority examples to even the balance.
• Oversampling — duplicating minority examples.
• Synthetic oversampling — creating new minority examples by interpolating between real ones.
• Class weight — telling the loss that a minority error costs more.
• Cost-sensitive learning — using the actual costs of each kind of error rather than an even weight.
• Threshold tuning — leaving the model alone and moving the decision point.

The mental model

Understand first what imbalance actually breaks, because it is not the model — it is two other things.

It breaks the metric: accuracy is dominated by the majority class, so the number stops carrying information, which the metrics topic covered. And it breaks the default threshold: a model trained on one-percent positives produces probabilities that are mostly small, so a cut at one half labels almost everything negative even when the ranking is excellent. Notice that neither of those requires touching the data. Very often the model is fine and only the reporting and the cut are wrong, which is why the fixes should be tried in this order.

First, fix the metric. Report precision, recall and the area under the precision-recall curve. Establish the majority-class baseline explicitly, so everyone can see what "99 percent accurate" is worth.

Second, tune the threshold. Take the model's ranking, sweep the threshold, and choose the point that meets the requirement — a recall target, a precision target, or the minimum expected cost given what each error costs. This is free, it changes nothing about training, and it solves a large share of imbalance problems outright. Try this before resampling anything.

Third, weight the classes. Most estimators accept a weight that multiplies the loss contribution of the minority class, and setting it inversely to class frequency tells the model that a missed positive costs more. This changes what the model optimises rather than what data it sees, so no information is invented or discarded. It is the right next step and often the last one needed.

Fourth, and only then, resample. Undersampling the majority is fast and throws away real data. Oversampling by duplication risks memorisation. Synthetic oversampling interpolates new minority points between existing ones, which works better than duplication and assumes the space between two minority examples is also minority — sometimes true, sometimes not, and in high dimensions the interpolated points can land in genuinely empty regions.

Then the trap, which is the most common way imbalanced-data results turn out to be fiction. Resample only inside the training fold, never before the split. Synthesise before splitting and points interpolated from a minority example end up in validation while their neighbours are in training — so the model has effectively seen the validation data, and the score is inflated in a way nothing will reveal until production. The mechanical defence is the pipeline from the tools topic: put the resampling step inside the pipeline so that cross-validation applies it to each training fold and never to the held-out one. And note the other half: never resample the validation or test set at all. Their whole purpose is to reflect the real distribution, and balancing them destroys exactly that.

Two more habits worth carrying. Keep the class balance in mind when splitting — stratified folds, so a fold does not end up with three positives. And report the confusion matrix, not only the summary metrics: raw counts of caught, missed and falsely flagged are what let somebody outside the project judge whether the model is useful, and they are much harder to misread than a single number.

What you should now be able to explain or do

Say what imbalance actually breaks, and note that neither thing requires touching the data. Give the four responses in order and say why the order is what it is. Tune a threshold against a stated requirement rather than accepting one half. Explain what class weights change and what they do not. Say what synthetic oversampling assumes. State the resampling rule for splits, both halves of it, and name the mechanism that enforces it. Say why the confusion matrix belongs in the report.

Check yourself