7.8 Decoder models: the GPT family

What this is and why it exists

Every generative language system you have used is built on one objective: predict the next token. It sounds too small to explain what these systems do, and internalising why it is not is the key intuition of the era. This topic is that objective, the causal masking that makes it trainable in parallel, the cache that makes generation fast, and the empirical relationships that decided how large these models became and on how much data.

The vocabulary

• Autoregressive — generating one token at a time, each conditioned on all the previous ones.
• Causal masking — preventing a position from attending to anything after it.
• Teacher forcing — training on the true previous tokens rather than the model's own output.
• KV cache — stored keys and values from earlier positions, reused rather than recomputed.
• Prefill — processing the prompt in one pass; decode — producing tokens one at a time.
• Scaling law — an empirical relationship between size, data, compute and loss.
• Compute-optimal — the size and data combination giving the best result for a fixed compute budget.
• Open-weight — a model whose parameters are published for others to run.

The mental model

The objective is the entire training signal. Take text, and at every position ask the model to predict what comes next; the correct answer is the token that is actually there. No labels, no annotation, no task definition — every document is millions of training examples, and the supply is whatever text exists.

Why does that produce general capability? Because predicting the next token well requires almost everything. To finish a sentence about a chemical reaction you need chemistry. To close a bracket correctly you need to have kept the structure of the code in mind. To continue a dialogue in character you need a model of who is speaking. To complete "the answer is" after a worked arithmetic problem you need to have done the arithmetic. The objective is trivial to state and arbitrarily hard to satisfy, and capability appears not because anybody asked for it but because it is instrumentally necessary for the prediction. That is the intuition worth internalising: nothing here was designed as a reasoning system; reasoning is what fitting the objective well enough demanded.

Causal masking is what makes training efficient. A generative model must not see the future, but you do not want to run it once per position. The mask hides everything after each position, so a single forward pass over a document trains every position at once, each predicting its own next token with no access to the answer. Get this mask wrong and training is spectacular and generation is nonsense, as the attention topic warned. Training also feeds the true previous tokens rather than the model's own predictions, which makes training stable and creates a mismatch with generation, where the model must build on its own output and its own mistakes.

The cache is why generation gets faster after the first token, and the reason is worth working through. Generation has two phases. Prefill processes the whole prompt in one pass, in parallel, which is fast per token. Then decoding produces one token at a time, and naively each new token would recompute attention over the entire sequence from scratch. But the keys and values for the earlier positions do not change — they depend only on tokens already fixed. So store them, and each new step computes only the new position's query, key and value, attending against the stored rest. Work per step drops from growing with the sequence to constant, which is what makes generation practical.

The cost moves to memory: the cache holds keys and values for every layer, every head and every position, and it grows with the length of the conversation and with how many requests are in flight. That memory is the binding constraint on how many conversations a server can hold at once — one of the main reasons serving these systems is an infrastructure problem, and the motivation behind attention variants that share keys and values across heads.

Scaling laws are why the models grew. Empirically, loss falls in a smooth and predictable way as parameters, data and compute increase — smooth enough that the result of a large training run can be estimated from small ones, which is what makes committing a very large compute budget a decision rather than a gamble.

The correction that reshaped practice came from asking what to do with a fixed compute budget: make the model larger, or train it on more data? The earlier generation had answered "larger", and the analysis showed they were substantially undertrained for their size — for a given budget, a smaller model trained on considerably more data reaches a lower loss. The practical consequences were immediate and are still with us. Model sizes stopped climbing as fast while training data grew. And a smaller model that performs as well is cheaper at every single inference, forever, which matters far more in total than the one-off training cost. A compute-optimal training budget and a serving-optimal model size are different questions, and once a model will serve many requests, training past the compute-optimal point to get a smaller model is straightforwardly worth it.

Open-weight families changed what is possible for everyone else. Several organisations publish their parameters, which means you can run a capable model on your own hardware. Four consequences matter. Data control: text never leaves your infrastructure, which resolves a whole class of regulatory and confidentiality problem. Cost at volume: hardware you own has a fixed cost regardless of request count. Adaptation: you can fine-tune on your domain, which is not generally possible through an interface. And permanence: nothing is deprecated beneath you.

The costs are real too — you operate the infrastructure, you carry the serving problem the cache section described, and the strongest available systems are not always among the published ones. Read the licence rather than assuming, because "open weights" covers a range of terms, some permissive and some carrying restrictions on scale, on use, or on training other models from the outputs. The families matter more than any particular release: releases are superseded constantly, while the choice between running your own and calling somebody else's is a stable architectural decision with the trade set out above.

What you should now be able to explain or do

State the objective and explain why it produces general capability. Say what causal masking makes possible in training and what teacher forcing creates a mismatch with. Describe prefill and decode, and explain what the cache stores and what it saves. Say where the cost moves and why it constrains serving. Explain what scaling laws describe and why predictability matters. State the compute-optimal correction and why serving economics push smaller still. Give the four advantages and the real costs of running published weights, and say why the licence needs reading.

Check yourself