5.4 Regularization

What this is and why it exists

Regularisation is the standard control for overfitting, and it is one of the few knobs in machine learning whose behaviour you can genuinely understand rather than tune blindly. Two forms do almost all the work, they do different things, and the difference is visible in the coefficients. It also has a prerequisite that people forget so consistently that it is worth stating before anything else: scale first.

The vocabulary

• Overfitting — fitting the noise in the training data, so the model does worse on new data.
• Penalty — a term added to the loss that grows as the weights grow.
• Ridge (L2) — a penalty on the sum of the squared weights.
• Lasso (L1) — a penalty on the sum of the absolute weights.
• Elastic net — a blend of the two.
• Shrinkage — the pulling of weights towards zero that a penalty causes.
• Sparsity — many weights being exactly zero, so the model uses fewer features.
• Prior — a belief about the weights held before seeing the data.

The mental model

Add to the loss a term that grows with the size of the weights, and the fit now has to earn every unit of weight it uses: a feature only gets a large coefficient if the reduction in error is worth the penalty. That is the whole idea, and everything else is which measure of "size" you choose.

Ridge measures size as the sum of squares. Because the penalty on a weight grows with the weight, the pressure to shrink is strongest on the largest weights and gentle near zero — so ridge pulls everything down smoothly and almost never to exactly zero. It is the right default when you believe many features each contribute a little, and it is particularly good with correlated features, which it treats even-handedly by splitting the weight between them rather than picking one arbitrarily.

Lasso measures size as the sum of absolute values. The penalty's pressure is constant all the way down, so a weight whose contribution is smaller than that pressure is pushed to exactly zero and stays there. That gives sparsity: the model selects features as a side effect of fitting, which is genuinely useful when you believe only a handful of features matter and you want a short model somebody can read. Its weakness is the mirror of ridge's strength — given several correlated features, it tends to keep one and zero the others, and which one it keeps can change with a small change in the data.

Elastic net blends the two, with one setting for the overall strength and another for the mix. It exists precisely for the correlated-and-sparse case: it selects, like lasso, but shares credit among correlated features, like ridge.

The strength itself is a genuine trade-off and it is chosen by cross-validation, not by taste. Too little and you have the unregularised model with its overfitting; too much and every weight shrinks towards nothing until the model predicts roughly the average for everybody. The useful picture is a path: sweep the strength across several orders of magnitude, watch the validation score rise and then fall, and take the value at the top — or, if you want a more conservative model, the strongest penalty whose score is still within noise of the best.

Then the framing that ties this to the probability module. A penalty is a prior on the weights. Ridge is the same thing as believing, before seeing the data, that the weights are drawn from a bell curve centred at zero; lasso corresponds to a more sharply peaked belief that puts real mass exactly at zero. Fitting with a penalty is then finding the most probable weights given both the data and that prior. That is not a metaphor — the arithmetic is identical — and it explains why the strength is a statement about how much you believe the data over your prior.

Finally, the prerequisite that makes or breaks all of it. Scale the features first, always. The penalty acts on the weights, and a weight's size depends on the units of its feature: the same relationship expressed in metres and in millimetres needs weights a thousand times apart, so an unscaled penalty punishes a feature for being measured in small units rather than for being unhelpful. Scale, then regularise — and note that the intercept is normally left out of the penalty, because shrinking it would drag every prediction towards zero rather than towards the average.

What you should now be able to explain or do

Say what a penalty does to the fit in one sentence. Explain why ridge shrinks smoothly and lasso produces exact zeros, from the shape of each penalty. Choose between them for two described situations, including a correlated-features case. Say what elastic net exists for. Choose a strength by sweeping and reading a validation curve. Explain the prior framing and what it says about the strength. State the scaling rule and what goes wrong without it.

Check yourself