5.5 Bias, variance and model capacity

What this is and why it exists

When a model is not good enough, there are only a few things you can do: get more data, build better features, or change the model's capacity. Guessing between them is expensive — more data is slow and sometimes useless, more capacity is fast and sometimes harmful. Bias and variance are the vocabulary that turns that guess into a diagnosis, and learning curves are how you read it.

The vocabulary

• Bias — error from the model being too simple to represent the real pattern.
• Variance — error from the model being so flexible that it follows the noise in whichever sample it saw.
• Irreducible error — the part nothing can fix, because the outcome genuinely is not determined by what you measured.
• Capacity — how much shape a model can take: more parameters, more depth, less regularisation.
• Underfitting — high bias: the model is wrong in the same way on training and new data.
• Overfitting — high variance: the model is right on training data and wrong on new data.
• Learning curve — training and validation error plotted against the amount of training data.
• Double descent — the observation that, past the point of exactly fitting the data, error can fall again.

The mental model

Total error breaks into three parts, and it is worth saying in words rather than notation: the error of a model is what it systematically gets wrong (bias, squared), plus how much its predictions bounce around when you retrain it on a different sample (variance), plus the noise that was never predictable. The first two trade against each other as you change capacity — simple models are consistently wrong, flexible models are inconsistently right — and the third is a floor.

The diagnosis is a gap. Look at training error and validation error together, and it is the relationship between them that tells you what to do.

Training error high, validation error about the same and also high: the model is underfitting. It cannot represent the pattern, and more data will not help — you would be giving more examples to something that cannot use them. The fixes are more capacity, better features, or fewer constraints: a more flexible model, an interaction term, less regularisation.

Training error low, validation error much higher: the model is overfitting. It has enough capacity and is using it on noise. Here more data genuinely does help, and so do the other three levers: stronger regularisation, fewer or better features, and a simpler model.

Training error low, validation error low and close: you are done with this diagnosis. Whether it is good enough is a different question, answered by the baseline from the framing topic.

Learning curves make the same reading visual and add one thing the pair of numbers cannot: whether more data is worth buying. Plot both errors against the number of training examples. Two curves that have flattened and converged, both at a high value, mean the model is the constraint — collecting more data will change nothing, and this is the plot to show the person asking for a bigger dataset. Two curves still approaching each other with a gap remaining mean more data is still buying accuracy, and you can even read roughly how much more it would take.

Two symptoms are worth recognising directly. A model that scores far better on training than on validation, especially with many features and few rows, is memorising. A model whose validation score is unstable across folds is telling you about variance too — the spread between folds is a variance measurement you get for free from cross-validation, and a large spread means the score you report is uncertain regardless of its average.

Then double descent, briefly and honestly. The classical picture says error falls as capacity grows, bottoms out, and rises again as the model begins to overfit. Modern very large models show a further stage: pushed well past the point of fitting the training data exactly, test error can fall again. That is real, it is an active area of study, and it is why "bigger is always worse past a point" is not a safe rule for very large models. What has not changed is the diagnostic habit: compare training with validation, plot the curve, and let the numbers say which lever to pull. The picture got more complicated; the measurement did not.

What you should now be able to explain or do

State the three parts of total error in words. Diagnose underfitting and overfitting from a pair of training and validation numbers, and name the fixes for each. Say when more data helps and when it cannot. Read a learning curve and decide whether to collect more data. Use the spread across folds as a variance measurement. Say what double descent complicates and what it leaves untouched.

Check yourself