1.3 Systems, rank, inverse, determinant

What this is and why it exists

Ax = b — find the inputs that produce these outputs — is the shape of fitting a linear model to data. This topic teaches when that question has no answer, one, or infinitely many, and what each case means for a model: too many constraints, exactly enough, or too much freedom. Rank is the number that tells you which world you are in, and conditioning tells you whether the answer you computed can be trusted.

The vocabulary

• Gaussian elimination — the systematic march to row-echelon form: subtract multiples of rows to zero out below the pivots; back-substitute.
• Rank — the number of genuinely independent rows (equivalently columns); the dimensions the map actually delivers.
• Column space — every b the map can produce; solvability means b lies in it.
• Null space — every x the map sends to zero; freedom means it holds more than the zero vector alone.
• Invertible — square, full-rank: every output comes from exactly one input, and the map can be run backwards.
• Pseudo-inverse — the honest stand-in when true inversion is impossible: the least-squares answer for unsolvable systems, the minimum-norm answer for underdetermined ones.
• Ill-conditioning — technically invertible but nearly not: tiny input noise becomes huge answer swings.
• Determinant — the factor by which the map scales volume; zero means a dimension was flattened away.

The mental model

Solvability lives in the column space; uniqueness lives in the null space. Ax is always a combination of A's columns — so a solution exists exactly when b is reachable by the columns. And if some nonzero x maps to zero, it can be added to any solution without changing the output — so solutions, when they exist, come in families. Two independent questions, two subspaces, and rank counts what is left after redundancy: full-rank square matrices answer both questions with "yes, exactly one".

The data-fitting translation is the reason this matters. Tall systems (more equations than unknowns — more data points than parameters) usually have NO exact solution; fitting means getting as close as possible, which is least squares (la6), delivered by the pseudo-inverse. Wide systems (more unknowns than data) have infinitely many exact solutions, and something must choose among them — the minimum-norm choice, or regularization; modern deep learning lives permanently in this overparameterised world. Recognising "no solution" versus "too many" as OVERDETERMINED versus UNDERDETERMINED fitting is the skill.

The determinant is the volume dial: feed the unit square through the map and the determinant is the area of what comes out (signed — negative means a flip). Zero determinant means the square was flattened into a line: a dimension destroyed, no way back, non-invertible. And NEAR-zero is the practical warning: ill-conditioning, where inversion technically succeeds and the answer is garbage because noise in b is amplified enormously. Real numerical code therefore avoids computing explicit inverses and determinants for solving — it factors and solves instead — but the concepts are how you diagnose why a fit exploded.

What you should now be able to explain or do

Row-reduce a small system and read its rank and solution count. Say where solvability and uniqueness each live. Classify a fitting problem as over- or underdetermined from its shape. Explain what a zero and a near-zero determinant each mean for inversion.

Check yourself