1.4 Eigenvalues and eigenvectors

What this is and why it exists

Some directions a matrix does not turn — it only stretches along them. Those are eigenvectors, the stretch factors are eigenvalues, and together they are a transformation's skeleton. This topic exists so that PCA — the workhorse of dimensionality reduction — becomes a theorem you can defend instead of an incantation, and so PageRank stops being a mystery.

The vocabulary

• Eigenvector / eigenvalue — a direction the map preserves, and the factor it stretches by: Av = λv.
• Eigendecomposition / diagonalization — rewriting a matrix as "change to the eigen-coordinate system, stretch each axis, change back".
• Spectral theorem — symmetric matrices have real eigenvalues and a full set of ORTHOGONAL eigenvectors; the family covariances and Hessians belong to.
• Positive (semi)definite — all eigenvalues positive (non-negative); the matrix shape of "a bowl" and of every legitimate covariance.
• Covariance matrix — variances on the diagonal, covariances off it; symmetric PSD by construction.
• Power iteration — multiply a vector by the matrix repeatedly, normalising; it converges to the dominant eigenvector.

The mental model

Eigenvectors are a transformation's grain — the directions along which its action is pure stretch. In eigen-coordinates a complicated matrix becomes diagonal: apply-the-map turns into scale-each-axis, powers of the matrix become powers of the eigenvalues, and long-run behaviour is read off the largest one. Diagonalization is exactly that change of viewpoint: rotate into the grain, stretch, rotate back.

The spectral theorem is why data people get the nice case. Covariance matrices are symmetric, so their eigenvectors form an orthogonal set — a genuine rotated coordinate system for the data cloud. Point the first axis along the direction of greatest variance (largest eigenvalue), the next along the greatest remaining variance at right angles, and so on: that is PCA, fully derived. The eigenvalues ARE the variances along the new axes, so "keep the components covering 95% of variance" is a statement about an eigenvalue sum. No hand-waving left: PCA is the spectral theorem applied to a covariance matrix. Positive semidefiniteness is the sanity floor — variance along any direction cannot be negative — and positive DEFINITE Hessians are how optimisation (ca3) recognises a bowl-shaped minimum.

Power iteration is the eigenvector you can compute with a for-loop: hit any starting vector with A again and again, normalising; the component along the dominant eigenvector outgrows all others and the iterate turns to point that way. PageRank is this at web scale — the link structure of the web as a giant matrix, its dominant eigenvector as the importance score of every page — history's most famous eigenvector computation.

What you should now be able to explain or do

Find eigenpairs of a 2×2 by hand and sketch their geometry. State the spectral theorem and what symmetry buys. Derive PCA's recipe from the covariance matrix in four sentences. Explain why power iteration converges and what PageRank computes.

Check yourself