1.5 SVD and low-rank approximation

What this is and why it exists

The singular value decomposition is the one factorization that exists for EVERY matrix — rectangular, singular, ugly — and it says something startling: every linear map is a rotation, then axis-aligned stretching, then another rotation. It is also the mathematics of compression: keep the biggest stretches, drop the rest, and you have the best low-rank approximation possible — the theorem under PCA, recommender systems, and LoRA fine-tuning.

The vocabulary

• SVD — A = UΣVᵀ: orthogonal, diagonal, orthogonal — rotate, stretch, rotate.
• Singular values — the diagonal of Σ, sorted largest first: the stretch factors, always non-negative.
• Truncated SVD — keep only the top k singular values and their vectors: a rank-k copy of A.
• Eckart–Young theorem — that truncation is the BEST rank-k approximation any method can produce, in the natural norms.
• Low-rank structure — when a big matrix's singular values decay fast, it is secretly a small matrix plus noise.
• LoRA — fine-tuning a large model by learning only a low-rank UPDATE to its weight matrices; the low-rank bet applied to adaptation.
• Condition number — largest singular value over smallest: the noise-amplification factor of la3's ill-conditioning, now with a formula.

The mental model

Rotate–stretch–rotate: V-transpose turns the input to align with the map's natural input axes, Σ stretches each axis by its singular value, U turns the result to the output's orientation. Nothing else is happening in ANY matrix — that is the whole zoo of linear maps tamed into three tame pieces. Singular values generalise eigenvalue magnitudes to rectangular matrices, and rank is the count of nonzero ones.

The compression story: write A as a sum of rank-one layers, each weighted by a singular value. Sorted largest first, the early layers carry the broad strokes and the late ones carry detail and noise. Truncating after k keeps the strokes — and Eckart–Young says no rank-k matrix whatsoever comes closer to A. That optimality is why the trick appears everywhere: PCA is truncated SVD of centred data (the right singular vectors ARE the principal components, computed more stably than via the covariance matrix); latent semantic analysis truncates a word–document matrix so that documents sharing MEANING land near each other even with no words in common; and a movie-ratings matrix truncates into user-taste and movie-flavour factors.

LoRA is the same bet placed on model adaptation: fine-tuning changes a giant weight matrix by an update that experience shows is approximately low-rank — so learn the update as a thin pair of matrices (rank 8 or 16 instead of thousands), cutting trainable parameters by orders of magnitude while keeping most of the adaptation. And the condition number falls out of the SVD for free: largest stretch over smallest stretch is exactly how much the map can amplify relative error — la3's warning, quantified.

What you should now be able to explain or do

Narrate A = UΣVᵀ geometrically. State Eckart–Young and what "best" means. Connect truncated SVD to PCA and to one application in words. Explain LoRA's low-rank bet in two sentences, and read a condition number off the singular values.

Check yourself