5.12 Bagging and random forests

What this is and why it exists

A random forest is what you get when you take the previous topic's unstable learner and average a great many of them. It is the strongest thing you can train in one command with almost no tuning, it is very hard to mistune badly, and it comes with a validation estimate for free. It is also where a specific and widespread misreading lives — the default feature importances, which are biased in a way nobody warns you about.

The vocabulary

• Ensemble — many models whose predictions are combined.
• Bootstrap sample — a sample of the same size drawn with replacement, so some rows repeat and some are absent.
• Bagging — bootstrap aggregating: fit a model per bootstrap sample and average.
• Decorrelation — deliberately making the members differ, so their errors do not coincide.
• Feature subsampling — considering only a random subset of features at each split.
• Out-of-bag — the rows a particular tree did not see, usable to score it.
• Impurity importance — the default measure, from how much each feature reduced impurity.
• Permutation importance — measured by shuffling a feature and seeing what the score loses.

The mental model

Averaging reduces variance, and the amount it reduces depends on how much the members disagree. Average ten identical models and you have one model; average ten that make different mistakes and the mistakes partly cancel while the shared signal survives. So the whole design problem is making the trees differ without making them bad, and a forest does it twice over.

Bootstrap sampling gives each tree a different training set, drawn with replacement, so roughly a third of the rows are missing from any given tree and others appear more than once. Feature subsampling goes further: at every split, only a random subset of features is considered. Without that second device, one strongly predictive feature would be chosen at the top of nearly every tree, and the trees would be near-copies whatever their samples — decorrelated data producing correlated models. Choosing among a random subset forces different trees to find different routes, and that is where most of the ensemble's advantage comes from.

The trees themselves are grown deep and left unpruned, on purpose. Individually they overfit; the averaging is what handles that. This is the opposite of the previous topic's advice and it is consistent: an unstable, low-bias learner is exactly what you want as an ingredient, because averaging removes variance and cannot remove bias.

Out-of-bag error is a pleasant consequence of the bootstrap. Each tree missed about a third of the rows, so each row can be predicted by the trees that never saw it, and averaging those gives an honest estimate with no separate validation split and no extra fitting. It is close to what k-fold would tell you, for free, and it is particularly useful when data is scarce. The one caution is the grouped-data caution from the cross-validation topic: bootstrap sampling is by row, so if several rows belong to one subject, out-of-bag is optimistic in exactly the way a random split would be.

Then the trap. The default feature importances are biased towards features with many possible split points. They are computed from how much each feature reduced impurity across the forest, and a feature with many distinct values — a continuous measurement, a high-cardinality category, an identifier — gets many chances to be chosen and to look useful, including when it is pure noise. The classic demonstration is adding a column of random numbers to a dataset and watching it place respectably in the importance ranking.

The remedy is permutation importance: shuffle one feature's values in the held-out data, re-score, and see how much the score drops. That measures what the model actually relies on rather than how often it was selected, and it is computed on data the model did not train on, which is the other half of the honesty. Two cautions come with it: with two correlated features, shuffling either one alone loses little because the other carries the same information, so both look unimportant — the fix is to consider them as a group; and importance is always relative to this model, not to the world, which is the same caution as the coefficients in the linear topic.

Practically, forests need very little tuning. More trees is monotonically better until it stops helping, so the count is a compute budget rather than a hyperparameter. The number of features considered per split is the one worth touching. And the honest comparison with the next topic: a forest is easier to get right and harder to break, while boosting will usually score better once tuned. Fit the forest first, always — it takes one command and it tells you what the problem is worth before you spend a day on gradient boosting.

What you should now be able to explain or do

Say why averaging reduces variance and what decides how much. Name the two devices that decorrelate the trees and say what would go wrong with only the first. Explain why the trees are grown deep on purpose, and why that is consistent with the previous topic. Describe out-of-bag error and name the situation where it is optimistic. Explain the bias in default importances, and describe permutation importance including its correlated-feature caveat. Say why a forest is the right first ensemble to fit.

Check yourself