6.16 Recurrent networks and their limits
You understand what attention replaced and why.
Before:05. Classical Machine LearningUnlocks:07. Natural Language Processing11. The ECE Bridge — Signals, Edge & Embedded AI
Recurrent networks process sequences step by step with carried state, and the LSTM and GRU gates were the fix for their vanishing gradients — for years, how sequence work was done. Attention has replaced them for most tasks, which is exactly why this topic matters: it is what attention replaced, and why. Streaming and low-latency niches still use recurrence, so the topic is context and a live tool at once, not history.
Work through these
RNN unrolling and BPTT
Processing a sequence one step at a time while carrying a hidden state, and training by unrolling that process. This was the standard approach to sequences before attention.
LSTM and GRU gates
Gated variants that let information pass through many steps without being crushed, which was the main fix for the failure below. They are still used where sequences are long and models must be small.
Bidirectional and stacked variants
Reading a sequence in both directions, and stacking several layers. Both are straightforward extensions worth recognising.
Long-range dependency failure
Even gated recurrent models struggle to connect information separated by long distances, and that limitation is precisely what attention removed. This is the item that motivates the next topic.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.