12.3 Long context and state-space models

What this is and why it exists

Attention compares every position with every other, so its cost grows with the square of the length. That single fact drives a whole research area, and this topic is the escape routes: an exact reformulation that removed most of the memory traffic, restrictions on what each position may attend to, an architecture that carries a compressed state forward instead, and hybrids of the two. It also carries the caution that matters commercially — a stated context length is a capacity, not a promise of recall.

The vocabulary

• Quadratic cost — growth with the square of the sequence length.
• Memory traffic — bytes moved between fast and slow memory.
• Exact reformulation — a different computation giving identical results.
• Sliding window — each position attending only to a nearby span.
• Sink token — a few early positions every position may always attend to.
• State-space model — a sequence architecture carrying a fixed-size state forward.
• Linear scaling — cost growing in proportion to length rather than its square.
• Recall at depth — whether a fact placed deep in a long input can be retrieved.

The mental model

Two costs grow with the square of the length, and they are not the same problem. The computation is the comparisons themselves. The memory is the score matrix — one number per pair of positions, written out and read back. On modern hardware the second is the binding one, which is the roofline point from the accelerator topic: attention at ordinary lengths is memory-bound, so the winning optimisation is to move less data rather than to compute less.

Which is exactly what the exact reformulation does. Rather than forming the whole score matrix in slow memory, process the sequence in blocks that fit in fast on-chip memory, computing and consuming the scores block by block and combining the results with a running normalisation so the answer comes out identical. It is not an approximation — the output is the same to numerical precision — and it removed most of the memory traffic, which is why longer contexts became practical on existing hardware without anybody giving anything up. When an exact method is available, take it before considering an approximation, and this is the case where one was.

Note what it does and does not change: the computation still grows with the square of the length, so it is a large constant-factor win rather than a change of order. The order is what the next two ideas address.

Restricting attention is the approximate route. In a sliding window, each position attends only to a nearby span, so cost becomes linear in length. Information still travels further than the window because stacking layers compounds the reach — the receptive-field argument from the convolution topic. The cost is that a genuinely long-range dependency has to survive several hops, and some do not.

A small refinement matters more than it sounds: keeping a few early positions attendable from everywhere. Models place a great deal of attention weight on the first tokens, and removing that ability when the window slides past them degrades quality sharply, so those positions are retained deliberately.

State-space models take the other route entirely: do not compare pairs at all. Carry a fixed-size state forward through the sequence, updating it at each step — which is the recurrent idea from the sequence topic, with two differences that make it work now. The update is designed so the whole sequence can be computed in parallel during training rather than step by step, removing the practical objection that killed recurrence. And the state's update is made input-dependent, so the model can decide what to keep and what to discard rather than applying the same fixed dynamics to everything.

The trade is exactly the one the recurrence topic named. Cost is linear in length and memory is constant during generation, which are large advantages. And everything must pass through a fixed-size state, so information that does not fit is lost — whereas attention keeps every position available and can reach any of them exactly. Attention is precise retrieval at quadratic cost; a carried state is compressed memory at linear cost, and that sentence is the whole comparison.

In practice the difference shows up as exactly that: these architectures are strong on long sequences and on throughput, and weaker on tasks needing exact recall of a specific detail from far back — which is a common task rather than an exotic one.

Hybrids are where much current architecture work sits, on the reasoning that the two mechanisms are good at different things. Interleave layers of each and you get the linear-cost bulk processing from one and the exact retrieval from the other, at a fraction of the attention layers a pure design would use. The results are good enough that this is a genuine direction rather than a compromise, and it is the thing to watch here rather than any individual model.

Then the caution that matters most commercially: a stated context length is a capacity, not a promise of recall. A model will accept that much input. Whether it uses all of it equally well is a separate question, and the answer is generally no — recall degrades with depth, material in the middle is attended to less reliably, as the token topic said, and methods that extend a context beyond the trained range degrade over the extension.

So test it, on your own material, before designing around it. Place a known fact at the start, at several depths in the middle, and near the end of a realistic input, and ask for it back. Twenty minutes gives you your actual working length, which is the number to build against.

And the practical comparison people actually need: long context or retrieval?

Long context wins when the material is a coherent whole that must be reasoned across — a contract, a codebase, a case file — where chunking would sever exactly the relationships you need. It is also far simpler: no store, no chunking strategy, no embedding model to maintain.

Retrieval wins on scale, on cost and on freshness. A corpus larger than any window has no long-context option at all. Retrieval sends only the relevant part, so it is cheaper per request by a wide margin at any volume. Documents can be updated without reprocessing anything, access control lives in the filter, and the answer can cite its sources.

Neither replaces the other, and the common production answer is both: retrieve the relevant documents, then use a long context to reason across all of them together. That gets the scale and the cost profile of retrieval with the coherence of a long window, and it is where the two ideas stop competing.

What you should now be able to explain or do

Distinguish the computational and memory costs of attention and say which binds. Explain what the exact reformulation changes and what it does not, and why an exact method is preferred. Describe sliding-window attention, why depth extends its reach, and why a few early positions are retained. Explain a carried-state architecture, the two changes that made it practical, and the trade against attention in one sentence. Say what hybrids are for. Test recall at depth on your own material. Choose between long context and retrieval by coherence, scale, cost and freshness, and combine them.

Check yourself