1.7 Derivatives, gradients, Jacobians, Hessians

What this is and why it exists

Training is turning millions of knobs to shrink one loss number, and the gradient is the object that says which way to turn every knob at once. This topic builds it honestly — partials, the gradient, the Jacobian for vector outputs, the Hessian for curvature — because every optimizer in ca4 and every line of backprop in ca2 is made of these.

The vocabulary

• Partial derivative — the sensitivity of the output to ONE input, others frozen.
• Gradient — all the partials stacked into a vector; points in the direction of steepest ascent.
• Directional derivative — the slope experienced walking in a chosen direction: the dot product of that direction with the gradient.
• Jacobian — for a vector-valued function, the matrix of every output's partial with respect to every input; the local linear map.
• Hessian — the matrix of second partials of a scalar function: curvature in every pair of directions.
• Stationary point — gradient zero: a minimum, a maximum, or a saddle — the Hessian tells which.

The mental model

Stand on a hillside blindfolded. Each partial derivative is the slope you feel probing one compass direction; the gradient assembles the probes into the direction of steepest CLIMB, and its negation — the direction the whole of deep learning walks — is steepest descent. The directional derivative formalises "slope in the direction I actually stepped": project the gradient onto your step (a dot product — la1's projection again). Perpendicular to the gradient, slope is zero: those are the contour lines on the map.

The Jacobian is what "derivative" becomes when the function outputs a vector: row i holds output i's gradient, and the whole matrix is the best LINEAR approximation of the function near the point — la2's transformations returning as local snapshots of nonlinear maps. A network layer's Jacobian is exactly what the chain rule (ca2) multiplies along.

The Hessian is the curvature ledger. At a stationary point the gradient is silent, and the Hessian speaks (la4's language): all eigenvalues positive — a bowl, a local minimum; all negative — a dome; MIXED signs — a saddle, downhill in some directions while uphill in others, the shape high-dimensional losses turn out to be full of. Its eigenvalues also say how STRETCHED the bowl is: a huge ratio between the steepest and gentlest curvatures (an ill-conditioned Hessian — la5's condition number, again) is a long narrow valley, and gradient descent zigzags across such valleys, which is the geometric reason momentum and Adam will exist in ca4.

The craft skill is computing a gradient by hand for something small — a two-variable quadratic, a tiny logistic loss — and CHECKING it numerically: nudge each input by a tiny h, watch the output move, compare to the partial. Gradient checking is five lines and has caught a generation's worth of derivation bugs.

What you should now be able to explain or do

Compute partials and assemble a gradient for a small function, and verify it numerically. Read a directional derivative as a projection. Say what the Jacobian is for a layer. Classify a stationary point from the Hessian's eigenvalue signs, and connect its conditioning to zigzagging descent.

Check yourself