6.2 Backpropagation by hand

What this is and why it exists

Backpropagation is the chain rule applied over a graph of small operations, and that is the entire algorithm. Deriving it once on paper for a two-layer network is the rite of passage of this module: afterwards, framework messages about shapes and graphs read as sentences rather than noise, and you know which layer to look at when a model trains slowly for no visible reason. The topic ends with the check that catches the errors your eyes will not.

The vocabulary

• Computational graph — the record of which operation produced which value, from inputs to loss.
• Forward pass — computing the loss.
• Backward pass — computing how the loss would change if each parameter changed.
• Local gradient — how one operation's output responds to its own input, in isolation.
• Chain rule — multiplying local gradients along a route to get the effect end to end.
• Accumulation — adding the contributions when a value was used in more than one place.
• Gradient checking — comparing your derivation against a numerical estimate.

The mental model

Start from what a gradient is for. Training adjusts every parameter a little in the direction that reduces the loss. To do that you need, for each parameter, the answer to one question: if this number increased slightly, would the loss go up or down, and by how much? Backpropagation computes all of those answers in a single sweep, at roughly the cost of one forward pass, which is the reason large networks are trainable at all.

The graph is the object. A forward pass is a sequence of small operations — multiply, add, apply an activation, compare to the target — and each one remembers what went into it. That record is a graph whose leaves are the inputs and whose root is the loss. PyTorch describes precisely this: autograd "records a graph recording all of the operations that created the data as you execute operations, giving you a directed acyclic graph whose leaves are the input tensors and roots are the output tensors." And a detail worth carrying forward: "the graph is recreated from scratch at every iteration, and this is exactly what allows for using arbitrary Python control flow statements, that can change the overall shape and size of the graph at every iteration."

The rule itself is one sentence. To find how the loss responds to something early in the graph, multiply together how each step responds to the one before it, all the way along the route. Every operation only needs to know two things: how its own output responds to its own input, and what came back from the operation after it. It multiplies them and passes the result further back. Nothing anywhere needs the whole picture, which is why this works on graphs of any size.

Accumulation is the part people get wrong. If a value was used by two later operations, it influences the loss along two routes, and its gradient is the sum of what comes back along each. Forget the addition and you have silently halved a gradient. This is also why frameworks require you to zero the gradients between steps: they add into the existing buffers by design, so leftovers from the previous batch are added to the current ones.

Derive the two-layer case, and keep the shapes straight. Work backwards from the loss: first how the loss responds to the output, then through the output activation, then to the last layer's weights and to its inputs, then through the hidden activation, then to the first layer's weights. Two facts make it manageable. A weight's gradient is the incoming signal from behind multiplied by the activation that entered it from in front — the two things that met at that multiplication. And the signal passed further back travels through the transpose of the weight rectangle, because it is moving in the other direction across the same connections. If you can say why a transpose appears there, you have understood the backward pass, and most shape errors in real code are that transpose in the wrong place.

Writing it in array code once changes how you read every framework afterwards. Implement a small network's forward and backward passes with plain array operations: no automatic differentiation, every derivative your own. It is an afternoon, and it converts the framework from magic into a labour-saving device whose error messages you can predict.

Then check it, because derivations are quietly wrong. Gradient checking compares your analytic gradient against a numerical one: nudge a single parameter up by a tiny amount, compute the loss, nudge it down by the same amount, compute the loss again, and the difference divided by twice the nudge estimates the derivative. Compare that against what your code produced, using a relative difference rather than an absolute one so the comparison means the same thing for large and small values. A sign error, a missing transpose or a forgotten accumulation shows up immediately. Two cautions: use a nudge that is small but not so small that floating-point subtraction destroys it, and switch off anything random — dropout in particular — or you are comparing two different functions. Check a handful of parameters rather than all of them; it is slow, and it is a check you run once rather than every step.

The payoff is that a mis-derived gradient does not announce itself. The model still trains, still improves, and settles somewhere mediocre — the exact failure the debugging topic is about. Twenty lines of finite differences rules it out in a minute.

What you should now be able to explain or do

Say what a gradient is used for and why one backward sweep gives all of them. Describe the computational graph and why it is rebuilt every iteration. State the chain rule in terms of what each operation needs to know. Explain gradient accumulation and why frameworks require zeroing. Derive the gradients for a two-layer network, and say why a transpose appears in the backward pass. Implement forward and backward passes in array code. Run a gradient check correctly, including the two cautions.

Check yourself