1.6 Norms, projections and least squares

Standard ML-mathematics theory — written August 2026

What this is and why it exists

Linear regression can be memorised as a formula or SEEN as geometry: the data's target vector cannot be reached, so land on the closest point you can reach — a projection. This topic derives that, names the machinery (norms to measure "close", normal equations and QR to compute it), and reframes regularization as a deliberate constraint on the projection. Derive it once from geometry and you can never quite forget it.

The vocabulary

Norm — a formal ruler for length: L2 (ordinary distance), L1 (sum of absolute values — the taxicab ruler), L-infinity (the largest coordinate), Frobenius (L2 for matrices).
Orthogonal projection — the closest point in a subspace to a given vector; the error is perpendicular to the subspace.
Least squares — choose parameters minimising the L2 norm of the residual — the sum of squared errors.
Normal equations — AᵀA x̂ = Aᵀb: perpendicularity of the residual, written as an equation.
QR factorization — A as an orthogonal Q times a triangular R; the numerically respectable route to the same x̂.
Regularization — an added penalty (ridge: L2, lasso: L1) that shrinks or sparsifies the solution.

The mental model

The picture: the columns of A span a subspace — every prediction the linear model can make lives in it. The target b, real and noisy, lies OFF that subspace (overdetermined, la3). The best possible prediction is the point of the subspace closest to b: drop a perpendicular from b onto it. That foot of the perpendicular is Ax̂, and the residual b − Ax̂ is orthogonal to every column of A. Writing that orthogonality down — Aᵀ(b − Ax̂) = 0 — and rearranging gives the normal equations. Linear regression, derived, with no calculus: it is a shadow falling on a subspace.

Which ruler you minimise with matters. L2 squares errors, so far-off points dominate — smooth, differentiable, sensitive to outliers. L1 counts errors linearly — robust to outliers, and with a taste for exact zeros. The same personalities recur when the norms penalise PARAMETERS: ridge (L2 penalty) shrinks all coefficients smoothly toward zero; lasso (L1 penalty) pushes some coefficients to EXACTLY zero, selecting features. Geometrically, regularization is a constrained projection: minimise the error subject to the coefficient vector staying inside a ball — an L2 ball is round, so the solution slides to small-but-nonzero coefficients; an L1 ball is a diamond, and solutions land on its corners, where coordinates vanish. The corner picture is the honest one-line answer to "why does lasso zero things out?".

Computation has a right answer too: forming AᵀA squares the condition number (la5), so serious code solves least squares through QR — factor A into orthogonal-times-triangular and back-substitute. Same x̂, far better behaviour; it is what the library call does beneath you.

What you should now be able to explain or do

Draw the projection picture and derive the normal equations from perpendicularity. Contrast L1 and L2 as rulers for errors and as penalties on parameters. Explain the ball-and-corner picture for ridge versus lasso. Say why QR beats the normal equations numerically.

Check yourself

State linear regression as geometry in two sentences.

The model's reachable predictions form a subspace spanned by A's columns; b lies off it. Least squares lands on the closest point — the orthogonal projection of b — making the residual perpendicular to the subspace.

Where do the normal equations come from?

From that perpendicularity: the residual must be orthogonal to every column, Aᵀ(b − Ax̂) = 0, which rearranges to AᵀA x̂ = Aᵀb.

Why does lasso produce exact zeros while ridge only shrinks?

The L1 constraint region is a diamond with corners ON the axes — optimal points hit corners, where coordinates are exactly zero. The L2 ball is round; solutions touch it where coefficients are small but nonzero.

Why does production code prefer QR to forming AᵀA?

AᵀA squares the condition number — noise amplification squared. QR reaches the same solution through orthogonal transformations that add no conditioning damage.

When would an L1 error (not penalty) be the better fitting ruler?

With outliers in the data: L2's squaring lets one wild point drag the whole fit; L1 counts it linearly and shrugs.

Go deeper

We haven't checked most of these for screen reader use yet.

Back to Norms, projections and least squares: work through the checklist