6.6 Regularization and augmentation

What this is and why it exists

There is a gap between how well your model does on data it trained on and how well it does on data it has not seen, and regularisation is the collection of deliberate ways to close it. The word "deliberate" is the topic. Every technique here costs something, and applying all of them at full strength at once produces a model that has stopped learning and a diagnosis you cannot make. Add one thing, measure, keep or discard.

The vocabulary

• Generalisation gap — training performance minus held-out performance.
• Dropout — randomly zeroing a fraction of activations during training.
• Weight decay — pulling parameters towards zero at every step.
• Early stopping — halting when held-out performance stops improving.
• Augmentation — transforming training examples to manufacture variety.
• Invariance — a change to the input that should not change the label.
• Mixup — blending two examples and their labels together.
• Label noise — deliberately imperfect targets.

The mental model

Read the gap before choosing anything. If training and validation performance are both poor, you are underfitting, and every technique in this topic makes it worse — you need more capacity, longer training, or better features. If training is excellent and validation is far behind, you are overfitting and this topic applies. Establishing which of those you have is the first step and it takes one plot.

Dropout switches off a random fraction of a layer's activations on every training step, so no unit can rely on any particular other unit being present, and the network is pushed towards representations that are spread out rather than concentrated. At evaluation time nothing is dropped and the activations are rescaled so their expected size matches. Two practical points: place it on the wide fully connected layers where the parameters are, not scattered through convolutional stacks where it interacts badly with normalisation; and note that it is used much less in modern architectures than it was, having been largely displaced by normalisation layers and heavy augmentation. Understand it, because it is everywhere in older code and it is a standard interview question, and reach for it after the cheaper things.

Weight decay pulls every parameter slightly towards zero at every step, which prefers models that spread their reliance across many small weights rather than a few large ones. It is nearly free, it is on by default in the optimiser you should be using, and the previous topic explains why the decoupled form is the one to want.

Early stopping is the cheapest regularisation available, because it is a decision rather than a mechanism: evaluate on held-out data each epoch, keep the best checkpoint, and stop when it has not improved for a set number of epochs. It requires a validation set, which is the actual cost — a set you must not also be selecting hyperparameters against, or the stopping point is chosen by the same noise it is meant to avoid. Set a patience rather than stopping at the first bad epoch, because validation curves are noisy and dip for a step or two routinely.

Augmentation is frequently the largest single improvement available, and it is the one to try first. Every transformation you apply is a statement that the label does not change under it, so the model is being told about an invariance you know and it does not. On images: flips, crops, rotations, colour and brightness changes, small distortions. On audio: time shifts, pitch and speed changes, added background noise, and masking bands of frequency or spans of time. On text it is harder, because most edits change meaning — synonym substitution and back-translation are the usual approaches, and both need checking.

The rule that makes augmentation safe is that the transformation must preserve the label in your problem. A horizontal flip is correct for photographs of animals and wrong for reading handwritten digits, where a mirrored digit is a different digit or no digit at all. Rotating a chest X-ray teaches the model something untrue about anatomy. And augment the training data only: an augmented validation set no longer measures performance on the data you will actually see.

Mixup and CutMix are the counter-intuitive pair. Mixup takes two training examples, blends their inputs in some proportion, and blends their labels in the same proportion — the model is asked to predict a mixture and cannot be certain about anything. CutMix instead pastes a rectangle of one image into another and mixes the labels by the area involved. Both improve robustness and calibration, and both feel wrong the first time because the blended images are not things that exist. That is the point: they force the model's behaviour between examples to be sensible rather than arbitrary, which is exactly the region where a confident wrong answer comes from. Deliberate label noise is the same idea applied to targets — a small amount makes the model less willing to memorise individual examples, and label smoothing from the loss topic is the well-behaved version of it.

Then the procedural rule this topic exists for. Stacking every technique at full strength produces a model that underfits, and you cannot tell which one did it, so people respond by weakening all of them uniformly and learn nothing either way. Add one thing at a time, keep the change if the validation curve improves, revert it if not, and record what you tried. In practice the order that pays is: fix the data, then augment, then weight decay and early stopping, and only then dropout or the blending methods. And if the gap is large and augmentation is available, augmentation is almost always the first move — it adds information about the problem, while every other technique on this list only removes capacity.

What you should now be able to explain or do

Read a training and validation curve and say whether regularisation is even the right response. Explain what dropout does in training and evaluation, where to place it, and why it is less common now. Say what weight decay prefers. Set up early stopping with patience and a validation set that is not doing another job. Choose augmentations for images, audio and text, and state the rule that decides whether one is safe. Explain what mixup and CutMix are doing and why the blended examples not existing is the point. Follow the one-at-a-time procedure and say why it matters.

Check yourself