12.3 Long context and state-space models
You know the alternatives to quadratic attention.
Before:08. Large Language Models
This topic maps the escape routes from quadratic attention: FlashAttention's exact speedups, sliding windows, state-space models like Mamba, and hybrids — with long context competing directly against retrieval for some workloads. It sits here because context economics shape system design. The marketing gap is measurable: a claimed context length is not usable recall, and models attend unevenly across very long inputs, so the advertised window and the working window differ.
Work through these
Efficient attention: FlashAttention, sliding window
Reformulations of attention that reduce memory traffic or restrict what each position attends to. These are what made longer contexts practical on existing hardware.
State-space models and Mamba
An alternative sequence architecture that carries a compressed state forward instead of comparing every pair of positions. It scales linearly, with different strengths from attention.
Hybrid architectures
Architectures combining both mechanisms, on the reasoning that each is better at something different. This is where much current architecture work sits.
Where long context beats retrieval, and where it does not
A long context and a retrieval system solve overlapping problems at different costs, and neither replaces the other. Knowing which your problem needs saves considerable expense.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.