5.5 Bias, variance and model capacity

Work through these

• The decomposition and the intuition

  Prediction error splits into error from a model too simple to fit the truth, error from sensitivity to the particular training sample, and irreducible noise. This decomposition is the diagnostic framework for the rest of the subject.

• Learning curves and what they show

  Plotting performance against training set size shows whether more data would help, which is a question worth answering before collecting any. The two curves converging or not is the whole reading.

• Underfitting vs. overfitting symptoms

  The symptoms differ clearly: a model failing on both training and held-out data is too simple, one succeeding only on training data is too complex. Distinguishing them decides what to do next.

• Double descent, briefly

  Very large models sometimes improve again past the point where the classical picture says they should get worse. It is covered briefly because it complicates the tidy story above.