7.6 The Transformer architecture in full

What this is and why it exists

You have the attention operation from the deep learning module. This topic assembles the rest of the architecture around it — what a block actually contains, how order gets into a system that has no notion of it, and what the encoder and decoder halves are each for. It ends with the cost that shapes the whole field: attention compares every position with every other, so cost grows with the square of the length, and nearly everything you read about long contexts is a response to that one fact.

The vocabulary

• Block — one repeated unit: attention, a per-position network, with normalisation and shortcuts.
• Encoder — a stack that reads a whole sequence with every position visible to every other.
• Decoder — a stack that produces a sequence, each position seeing only what came before.
• Cross-attention — a decoder layer attending to the encoder's output.
• Positional encoding — information about order, added because attention has none.
• Feed-forward block — a small network applied to each position separately and identically.
• Pre-norm and post-norm — normalising before or after the sub-layer.
• Quadratic cost — growth with the square of the sequence length.

The mental model

A block is two sub-layers, each wrapped the same way. First multi-head attention, letting positions exchange information. Then a feed-forward network applied to each position independently — the same small network everywhere, typically widening to several times the model's dimension and narrowing back. Each sub-layer sits inside a residual connection and a normalisation. Stack that block a few dozen times and you have the architecture.

The division of labour between the two sub-layers is the clearest way to hold it. Attention is where positions talk to one another; it moves information across the sequence and mixes nothing within a position that was not gathered from elsewhere. The feed-forward network is where each position is transformed on its own; it never looks sideways. Mixing across positions, then processing within positions, alternating — that is the whole design, and it is worth being able to say in one sentence. Most of the parameters live in the feed-forward blocks, which surprises people who assume attention dominates because it gets the attention.

The original form has both halves. An encoder stack reads the input with no masking, so every position sees every other in both directions, producing a representation of the whole sequence. A decoder stack produces output one position at a time, with causal masking so a position never sees the future, and with a cross-attention sub-layer in each block where the queries come from the decoder and the keys and values from the encoder's output. That is how a translation model consults the source sentence while writing the target.

Modern systems mostly keep one half. Encoder-only stacks are for understanding tasks — classification, extraction, ranking — where you have the whole text and want a representation of it. Decoder-only stacks are for generation, and they are what the current generative systems are. Encoder-decoder remains natural for genuine sequence-to-sequence transformation. Knowing the full form is what makes each variant readable: when you meet a description of a stack, ask whether it masks, and whether it has cross-attention, and you know which it is.

Positional encoding is not a detail, and treating it as one is the trap. Attention computes a weighted blend over positions and the operation is indifferent to their order — shuffle the input and it computes the same set of blends. Without position information a transformer sees a bag of tokens, so word order would be as invisible as it was in the counting topic. Position must be injected deliberately, and how it is injected decides how the model behaves on lengths it did not train on.

Four schemes, in the order they arrived. Fixed patterns built from sine and cosine waves of different frequencies, added to the input embeddings; they need no parameters and in principle extend to any length. Learned position embeddings, a vector per position trained with the model; simple and effective, and they cannot represent a position beyond the longest seen in training, full stop. Rotary encoding, which rotates the query and key vectors by an angle proportional to position, so the attention score between two positions depends on their separation rather than their absolute places — a relative encoding built into the attention operation itself, and the reason it extends well is that the relationship it encodes is the same everywhere in the sequence. Attention with a linear distance penalty, which adds a penalty to the attention score proportional to how far apart two positions are, biasing each head towards nearby context by a fixed amount and extending to longer sequences without modification.

The last two are why long contexts became feasible, and they are the reason a model can sometimes be extended past its training length: the position information is relative, so nothing about a longer sequence is entirely unfamiliar. It is worth knowing which scheme a system uses before assuming anything about its behaviour on long inputs.

Normalisation placement genuinely changes training stability. The original arrangement normalises after each sub-layer, inside the residual path, which means the residual stream is renormalised at every block; deep stacks trained this way need careful warmup and are temperamental. Normalising before each sub-layer instead leaves a clean residual path from the input all the way to the output, gradients flow along it without repeated rescaling, and very deep stacks train stably with much less warmup. That is why the second arrangement became standard. If you are reproducing a result, copy the placement exactly — it is one of the few architectural details that changes whether training works at all.

Then the cost that shapes everything. Every position attends to every other, so the number of score computations grows with the square of the sequence length, as does the memory to hold them. Double the length and the attention cost quadruples. This one fact drives the amount of research on long context: sparse patterns where each position attends to a subset, low-rank approximations, memory-efficient implementations that never materialise the full score matrix, and recurrent-flavoured designs that borrow from the family attention displaced. It is also why context length is a headline property of a system rather than an implementation detail, and why the tokenizer topic's point about how text splits has a direct cost consequence.

What you should now be able to explain or do

Draw a block and name every part. State the division of labour between the two sub-layers in one sentence, and say where most parameters live. Describe the encoder and decoder stacks and what cross-attention connects. Identify which variant a described stack is by asking two questions. Explain why position must be injected and what happens without it. Compare the four positional schemes and say why the relative ones extend better. State the effect of normalisation placement and why to copy it. Explain the quadratic cost and what it drives.

Check yourself