6.16 Recurrent networks and their limits

What this is and why it exists

Recurrent networks were how sequence work was done for years, and understanding them is not a matter of respect for history. They explain precisely what attention removed, which is the best possible preparation for the next topic. They are still the right answer in streaming and low-latency settings, where you cannot hold the whole sequence. And the question of why long-range dependencies defeated them is asked in interviews because the answer demonstrates whether you understand gradients or only APIs.

The vocabulary

• Hidden state — a vector carried from one step to the next, holding what has been seen.
• Unrolling — laying the repeated computation out as a long chain, one copy per step.
• Backpropagation through time — running the backward pass along that chain.
• Gate — a learned value between zero and one that controls how much passes.
• Cell state — a second carried vector in the gated design, changed mostly by addition.
• Bidirectional — reading the sequence forwards and backwards and combining.
• Truncation — cutting the backward pass short after a fixed number of steps.

The mental model

A recurrent network is one small network applied over and over. At each step it takes the current input and the hidden state from the previous step, and produces a new hidden state. The same weights are used at every step — the sequence equivalent of the parameter sharing convolutions use across space — so the network handles a sequence of any length with a fixed number of parameters. The hidden state is the entire memory of everything seen so far, compressed into a fixed-size vector.

Training unrolls it. Lay out one copy of the network per time step and you have a very deep feedforward network whose layers happen to share weights. Backpropagation runs along that chain from the end to the beginning, and every weight's gradient is the sum of its contributions at every step. For long sequences this is expensive in memory, since every step's activations must be kept, which is why truncation — running the backward pass only a fixed number of steps back — is standard.

And now the failure, which is the whole point of the topic. That chain multiplies a factor at every step, exactly as depth did in the activations topic, except that here the factor is the same one repeated because the weights are shared. Repeated multiplication of anything below one collapses towards zero and anything above one grows without bound. So a gradient arriving from step two hundred back is either nothing or a catastrophe, and in practice it is nothing: the network can learn what happened five steps ago and cannot learn what happened two hundred steps ago. The information is not blocked in the forward pass; the learning signal cannot reach back far enough for the network to discover it is useful. Exploding gradients are the easier half of this and gradient clipping handles them. Vanishing gradients are the hard half, and they are what the gates were invented for.

Gated designs fix it by changing how the carried state is updated. Instead of replacing the state with a transformed version at every step, the long short-term memory design keeps a cell state that is modified mostly by addition, with learned gates deciding what to forget, what to add and what to expose as output. Because the cell state passes forward with little multiplication, a gradient can travel back along it without being repeatedly scaled — an uninterrupted route through time, which is the same trick the residual connection uses through depth. The gated recurrent unit is a simplified version with fewer gates and no separate cell state; it trains faster, uses less memory, and performs comparably on most tasks. Choose the simpler one by default and the more complex one if it measurably helps.

Two extensions worth recognising. Bidirectional models run one recurrence forwards and another backwards and combine the two hidden states, so each position is informed by the whole sequence. This helps a great deal for labelling and classification, and it is impossible for generation or for streaming, because the backward pass needs the future. Stacking several recurrent layers, with each reading the previous one's outputs, adds depth in the ordinary way and is straightforward.

Then the honest assessment. Gates made long-range learning possible rather than solved. A gated model still struggles to connect information hundreds of steps apart, because everything must pass through a fixed-size vector one step at a time — the entire past is compressed into that vector, and what does not fit is lost. There is a second problem alongside it: the computation is inherently sequential. Step one hundred cannot be computed until step ninety-nine is done, so a recurrent model cannot use a parallel accelerator well, and training is slow in a way no hardware fixes.

Attention removes both at once, which is why it displaced this family so completely. Every position can look directly at every other position, so the route between two distant points is one step rather than hundreds, and there is nothing to vanish along. And because the positions do not depend on one another sequentially, the whole sequence is computed in parallel. That is the argument in full, and it is worth being able to give it in two sentences.

Where recurrence survives: streaming and low-latency work, where input arrives one item at a time and a fixed-size state that updates in constant time per step is exactly right, while attention over a growing history is not. Small models on constrained hardware, where the memory cost of attention over a long sequence is prohibitive. And very long sequences, where the cost of attending from every position to every other becomes the binding constraint — a live area of work that keeps borrowing from this family.

What you should now be able to explain or do

Describe what a recurrent step computes and what the hidden state holds. Explain unrolling, backpropagation through time, and why truncation is standard. Give the repeated-multiplication argument for vanishing gradients, and say precisely what is lost. Explain how a gated cell state creates an uninterrupted route for gradients, and compare it to a residual connection. Choose between the two gated designs sensibly. Say what bidirectional models buy and where they cannot be used. Give the two-sentence argument for why attention replaced recurrence, and name where recurrence still wins.

Check yourself