8.1 How LLMs are trained
You can explain pretraining, SFT and preference tuning to a non-specialist.
Before:07. Natural Language ProcessingUnlocks:09. Agentic AI12. Frontier Topics
The LLM production line has three stages: pretraining on web-scale text, supervised fine-tuning for instruction-following, preference optimisation for behaviour — and knowing which stage shaped which property tells you what prompting can and cannot fix. It opens the LLM module because everything later assumes this map. The attribution mistake is common: capabilities mostly come from pretraining, behaviour mostly from the later stages, and confusing the two leads to fixing the wrong layer.
Work through these
Pretraining corpora, compute and scaling laws
The first stage learns from an enormous body of text, and how much data and computation to use for a given size is now guided by empirical relationships. Understanding this stage explains what these models fundamentally are.
Supervised fine-tuning and instruction data
The second stage teaches the model to follow instructions, using curated examples of requests and good responses. This is what turns a text predictor into an assistant.
RLHF, PPO, DPO and preference optimization
The third stage tunes the model toward responses people prefer, using several named methods that differ in complexity. It is where much of the perceived quality difference comes from.
Reasoning training and verifiable rewards
More recent training approaches reward reaching a verifiably correct answer, which suits mathematics and code. This is the newest of the stages and the one changing fastest.
Sign in to keep your progress.
Free resources
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.