1.8 Chain rule and computational graphs

What this is and why it exists

Backpropagation — the algorithm every deep learning framework runs — is the multivariable chain rule organised over a graph, plus one economic decision about direction. Hand-derive it once for a small network and autograd stops being magic forever; that is this topic's entire mission.

The vocabulary

• Chain rule — the derivative of composed functions is the product of the pieces' derivatives; in many variables, contributions along every path are summed.
• Computational graph — the function drawn as nodes (operations) and edges (data flow); what frameworks literally build.
• Local gradient — each node's derivative with respect to its own inputs, knowable without seeing the rest of the graph.
• Forward mode — accumulate derivatives from inputs toward outputs: one pass per INPUT.
• Reverse mode (backprop) — accumulate from outputs back toward inputs: one pass per OUTPUT.
• Autograd — the machinery that records the graph as your code runs and replays it backwards for gradients.

The mental model

Sensitivities multiply along a path. If a nudge to x moves u by a factor of 3, and that move in u moves the loss by a factor of 2, then x moves the loss by 6: derivative times derivative along the chain. When x reaches the loss through SEVERAL paths, each path multiplies internally and the paths ADD. That is the whole multivariable chain rule: multiply along paths, sum over paths — and a computational graph is the bookkeeping device that makes the paths visible.

Each node needs only its local gradient: addition passes sensitivity through unchanged; multiplication passes each input the OTHER input's value; the maximum routes everything to the winner. Complex derivatives assemble themselves from these penny-simple pieces — nobody differentiates the whole expression, which is why autograd can handle any program you write.

The direction decision is pure economics. Forward mode answers "how does everything depend on THIS input?" — one sweep per input. Reverse mode answers "how does THIS output depend on everything?" — one sweep per output. Training has millions of inputs (parameters) and ONE output (the scalar loss): reverse wins by a factor of the parameter count. One forward pass computes and stores the intermediate values; one backward pass sweeps loss-sensitivity from the end to every parameter. That stored forward state is also the memory bill of training — the reason batch size fights model size for GPU memory, and the thing checkpointing trades recomputation for.

The rite of passage: a two-layer network on paper — linear, sigmoid, linear, squared error. Draw the graph, write each node's local gradient, multiply-and-sum backwards to every weight, then check numerically (ca1). It takes an evening and permanently converts "backprop" from incantation to bookkeeping.

What you should now be able to explain or do

State the chain rule as multiply-along-paths, sum-over-paths. Draw a small function as a graph and annotate local gradients. Explain in one sentence why training uses reverse mode. Hand-derive and numerically verify a tiny network's gradients.

Check yourself