8.1 How LLMs are trained

What this is and why it exists

There is a production line with distinct stages, and each stage put a different property into the model you use. Knowing which stage produced which behaviour tells you what prompting can fix and what it cannot — a system prompt cannot undo a preference tuned into the weights, and no amount of instruction adds knowledge the pretraining never saw. This topic is that line, stated so you could explain it to somebody who has never opened a terminal.

The vocabulary

• Pretraining — learning to predict the next token over an enormous body of text.
• Base model — the result of pretraining alone, before any instruction training.
• Supervised fine-tuning — training on curated pairs of request and good response.
• Preference optimisation — tuning toward responses people prefer over alternatives.
• Reward model — a model trained to score responses the way people would.
• Reasoning training — rewarding a verifiably correct final answer.
• Verifiable reward — a signal that can be checked mechanically, such as a passing test.
• Alignment tax — capability lost in exchange for better behaviour.

The mental model

Stage one, pretraining, is where the knowledge and the capability come from. The objective is the one from the decoder topic: predict the next token, over an enormous corpus. There are no labels, so the data can be as large as you can gather and clean, and cleaning is a much larger part of the work than people expect — deduplication, quality filtering, removing the same benchmark you plan to be evaluated on. How large the model should be and how much text it should see, for a given compute budget, is guided by the empirical scaling relationships, and the correction those produced is that earlier models were undertrained for their size.

What comes out is a base model, and a base model is not an assistant. Ask it a question and it may continue with more questions, because a page of questions is a plausible continuation of a question. It has the knowledge and the capability, and it has no idea that a request is meant to be answered. That gap is the whole reason the later stages exist, and it is the clearest way to see what each stage contributes.

Stage two, supervised fine-tuning, turns a text predictor into an assistant. Train on curated examples of a request paired with a good response, and the model learns the shape of the interaction: a question is answered, an instruction is followed, an answer ends rather than continuing into an imagined next question. The data is small by pretraining standards — thousands to hundreds of thousands of examples, written or curated by people — and its quality dominates its quantity. This stage teaches form and behaviour, not facts, which is the same distinction the fine-tuning topic later makes for your own work.

Stage three, preference optimisation, is where most of the perceived quality difference lives. The problem it solves is that "good response" is easier to recognise than to write: people can reliably say which of two answers is better and struggle to compose the ideal one. So collect comparisons rather than examples.

The classical route trains a reward model on those comparisons — a model that scores a response the way the human raters did — and then tunes the language model to score well under it, using a reinforcement-learning method with a constraint keeping it from drifting far from where it started. It works and it is machinery: a second model, an unstable training loop, and a reward model that can be gamed by outputs that score well and are bad, which is the characteristic failure.

The direct-preference route removes the middle. It shows, with an algebraic argument, that the optimal policy under such a reward has a closed form, so the preference data can be used to train the language model directly with an ordinary supervised loss and no separate reward model and no reinforcement loop. It is far simpler to run, which is why it became widespread, and the two approaches remain in use for different situations.

This is the stage to attribute correctly, and misattributing it is the trap this topic exists for. Refusal behaviour, tone, hedging, the willingness to say "I don't know", the formatting habits, the reluctance around certain topics, the tendency to be agreeable — almost none of that came from pretraining. It was tuned in, by choices somebody made about what raters preferred. Which means it is a property of the product rather than of the technology, it differs between systems for reasons that are decisions rather than discoveries, and prompting can bend it a little and not remove it.

Stage four, reasoning training, is the newest and the one moving fastest. The idea is that for tasks where correctness can be checked mechanically — mathematics with a known answer, code that passes tests, a proof a checker accepts — you do not need human preference at all. You need a verifiable reward: reward the model for reaching an answer that checks out. Trained this way, models learn to spend more computation before answering, working through intermediate steps, checking and revising.

Two things follow that are worth carrying. This only applies where correctness is checkable, so it strengthens mathematics, code and formal tasks much more than open-ended writing or judgement. And it changes the economics of using these systems: producing a better answer by thinking longer means more tokens and more latency per request, so the trade between quality, cost and speed becomes something you choose per request rather than a fixed property.

Finally, what this tells you about your own work. Prompting operates on the last stages, not the first: it can shape form, tone and the framing of a task, and it cannot install knowledge or remove a tuned preference. Missing knowledge is a retrieval problem. Wrong form is a prompting or fine-tuning problem. Behaviour you dislike that survives clear instruction is preference tuning, and it is the vendor's decision, not a bug in your prompt. Diagnosing which of those you have saves the most time of anything in this module.

What you should now be able to explain or do

Name the four stages and say what each contributes. Explain why a base model is not an assistant. Say what makes supervised fine-tuning about form rather than facts. Explain why preference data is collected as comparisons, and describe both routes and their trade. Attribute refusal, tone and hedging to the right stage and say what follows for prompting. Describe reasoning training, what a verifiable reward is, and where it does and does not apply. Diagnose whether a problem you are having is knowledge, form or preference.

Check yourself