7.8 Decoder models: the GPT family
You understand causal language modelling as the base of everything current.
Before:06. Deep LearningUnlocks:08. Large Language Models
Decoder models predict the next token behind a causal mask — the deceptively simple objective underneath GPT, Llama and everything currently called an LLM — with the KV cache making generation fast and scaling laws explaining why the models grew. It pairs with the encoder topic as the other half of the family. The mistake is underestimating the objective: predicting the next token well enough forces a model to absorb grammar, facts and reasoning patterns, which is the whole surprise of the era.
Work through these
Autoregressive generation
Generating one token at a time, each conditioned on everything produced so far. This is the mechanism behind every current generative language model.
Causal masking and KV cache
Preventing a position from seeing later positions, and caching earlier computations so generation does not redo work. The cache is why generation gets faster after the first token.
Scaling laws and the Chinchilla result
Empirical relationships between model size, data size and performance, and the result showing that earlier models were trained on too little data for their size. These shaped how models are built now.
Open-weight families: Llama, Mistral, Qwen, Gemma
Several families of models whose weights are published, which is what makes local and private deployment possible. The families matter more here than any particular release.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.