5.11 Decision trees

What this is and why it exists

A decision tree is the most explainable model you will ever train: a prediction is a path of questions you can read aloud to a manager, and they will understand it. That property is worth a great deal in any setting where somebody must be told why. The topic also ends by explaining why single trees are rarely the final answer — which is the reason the next two topics exist.

The vocabulary

• Node — a question about one feature; leaf — a final prediction.
• Split — the question and the threshold or category set it uses.
• Gini impurity — how mixed a node's classes are; zero when a node is pure.
• Entropy — the same idea measured in bits of uncertainty.
• Information gain — the reduction in impurity a split achieves.
• Depth — how many questions deep the tree may go.
• Leaf size — the fewest examples a leaf may contain.
• Pruning — cutting back a grown tree where the extra splits do not pay.

The mental model

Growing a tree is greedy and simple: at each node, try every feature and every reasonable split point, score each by how much it reduces impurity, take the best, and repeat on both sides until a stopping rule fires. Greedy means it never reconsiders — the split chosen at the top is fixed, even if a different first question would have allowed a much better tree — which is one reason a single tree is not the last word.

The splitting criteria are two measures of "how mixed is this group". Gini is the probability that two items drawn from the node have different labels; entropy measures the uncertainty in bits. Both are zero for a pure node and largest for an even mixture, and in practice they choose almost the same splits, so the choice between them is not where your attention should go. For regression, the same machinery uses squared error: choose the split that most reduces the variance of the target within the two sides.

What does matter is when to stop, because an unconstrained tree will keep splitting until every leaf holds one example — a perfect memory of the training data and nothing else. The controls are depth, minimum leaf size, and a minimum improvement required for a split. The most reliable of these is the minimum leaf size: it directly states how many examples a prediction must rest on, which is a statement you can defend to somebody. A leaf built from three rows is not a rule, it is an anecdote. Pruning is the other approach — grow it fully, then cut back the branches whose contribution does not survive validation — and it often produces better trees than stopping early, because a split that looks poor can enable a good one beneath it.

Trees handle two things gracefully that most other models do not. Categorical features need no numeric encoding in principle: a split can ask which categories go left. And missing values can be handled directly, by sending them down whichever side works better or by learning a default direction. Not every implementation does either — some still require you to encode categories yourself — but the model has no objection, which is a real practical advantage over the distance- and gradient-based methods.

They also need no scaling at all, for a reason worth stating: a tree splits on order, not on magnitude. Asking "is this feature above 5.2" gives the same partition whichever units the feature is in. Everything in this module that demanded scaling did so because it measured distances or took gradients; trees do neither.

Then the honest ending. A single deep tree is unstable. Change a handful of training rows and the top split can change, and everything below it changes with it — so two trees fit on samples from the same data can look entirely different while performing similarly. That instability is the same thing as high variance, and it means a tree's exact structure should not be read as the structure of the world. It is also, precisely, what averaging fixes: an unstable learner with roughly the right idea is the ideal ingredient for an ensemble, which is where the next two topics begin.

So the practical position: use a shallow tree when explainability is the requirement and you can accept a modest score, and use trees inside an ensemble when the score is the requirement. And when you show somebody a tree, show a shallow one — a tree of depth twenty is no more interpretable than any other model, and claiming otherwise is where the "interpretable model" argument is usually lost.

What you should now be able to explain or do

Describe how a tree is grown and say what "greedy" costs. Say what Gini and entropy measure and why the choice between them rarely matters. Name three stopping controls and say which is the most defensible, with the reason. Explain why pruning can beat early stopping. Say why trees need no scaling, and connect it to what other models needed it for. Explain instability and why it makes trees good ensemble ingredients. Read a prediction path aloud as an explanation.

Check yourself