1.2 Matrices as linear maps

What this is and why it exists

A matrix times a vector looks like an arithmetic drill and is actually a machine: the matrix TRANSFORMS the vector — rotates, stretches, projects. Every layer of a neural network is exactly such a machine (plus a nonlinearity), so this reframe from rule to transformation is the single most load-bearing idea in the module.

The vocabulary

• Linear map — a transformation that respects adding and scaling: lines stay lines, the origin stays put.
• Matrix–vector product — apply the map; equivalently, take a combination of the matrix's columns with the vector's entries as weights.
• Composition — apply one map after another; its matrix is the PRODUCT of the two matrices, which is why matrix multiplication exists and why order matters.
• Diagonal matrix — stretches each axis independently; the cheapest transformation there is.
• Orthogonal matrix — pure rotation or reflection; lengths and angles preserved; its transpose is its inverse.
• Symmetric matrix — equal to its transpose; the well-behaved family (covariances, Hessians) with the beautiful eigentheory of la4.
• Sparse matrix — mostly zeros, stored and multiplied by its nonzeros only; how anything at web scale is possible.
• Broadcasting — NumPy's rule for stretching mismatched shapes to fit an elementwise operation.

The mental model

Watch what a matrix does to the unit square. Feed the two standard basis vectors through the map: where they land are exactly the matrix's COLUMNS, and the whole square follows — sheared, rotated, squashed. "The columns are where the basis goes" reads any small matrix at sight: a diagonal matrix moves each basis vector along its own axis (pure stretch); an orthogonal matrix swings them around rigidly (rotation); a matrix with dependent columns flattens the square into a line — dimensions lost, a preview of rank.

Multiplication is composition, and that explains its every quirk. "Rotate then stretch" versus "stretch then rotate" end differently, so AB is not BA. Undoing "A then B" means undoing B first, so the inverse of AB is B-inverse times A-inverse. Nothing to memorise — everything follows from reading products as "do this, then that". A deep network is precisely a long composition: matrix, nonlinearity, matrix, nonlinearity — and without the nonlinearities the whole stack would collapse by this very rule into ONE matrix, which is why the nonlinearities must be there.

Broadcasting is the same story in code: subtracting a row of column-means from a whole data matrix works because NumPy stretches the row across every row of the matrix. The rule: align shapes from the right; each pair of dimensions must match or one of them must be 1, and the 1 is stretched. Say "this row is applied to every row" in words before trusting the code — broadcasting errors are shape errors you did not hear yourself make.

What you should now be able to explain or do

Predict a small matrix's action by reading its columns. Explain why multiplication composes and why order matters. Recognise diagonal, orthogonal, symmetric and sparse on sight and name each one's superpower. Predict the result shape of a broadcast operation before running it.

Check yourself