12.1 Reasoning models and test-time compute

What this is and why it exists

Some systems now spend much more computation at answer time — working through steps, trying several attempts, checking their own work — and buy accuracy on hard problems with it. That spend is a product decision with a latency price and a cost price, and the decision is per task class rather than global. Defaulting to it everywhere makes routine work slower, more expensive, and occasionally worse.

The vocabulary

• Test-time compute — computation spent producing an answer, as against training.
• Chain of thought — intermediate reasoning produced before the answer.
• Trained reasoning — that behaviour learned during training rather than requested.
• Sampling several attempts — generating multiple candidates for one question.
• Self-consistency — taking the most common answer among them.
• Verifier — something that checks a candidate answer.
• Process supervision — rewarding the intermediate steps, not only the final answer.
• Overthinking — spending more computation and arriving somewhere worse.

The mental model

The idea underneath is one sentence: quality can be bought at answer time, not only at training time. For decades the way to a better model was a bigger model trained on more data — a cost paid once, then amortised over every request. The newer axis is to spend more per request instead, and the two are genuinely different: training compute is capital, inference compute is operating cost, and an improvement that costs you on every single request has completely different economics from one that does not.

Chain of thought as trained behaviour is what distinguishes these systems from a prompt technique. Asking a model to think step by step was always available and it helped. What changed is training the behaviour in: the model produces its reasoning by default, at length, without being asked, and — the part that matters — it was trained against whether the final answer was verifiably correct, so the reasoning was shaped by what actually leads to right answers rather than by what reads like reasoning.

That is why it is not the same thing as the prompt technique. It also means asking such a model to think step by step is redundant, and occasionally counterproductive, because it is already doing that and the instruction interferes with how it was trained to do it.

Test-time scaling has three mechanisms and they compose.

Think longer: produce more intermediate reasoning before answering. Helps on problems with steps; helps nothing on retrieval or on formatting.

Sample several attempts: generate the answer more than once and combine. With a definite answer, take the majority — several independent attempts agreeing is real evidence, and it lifts accuracy on mathematics and code noticeably. With no definite answer there is no majority to take, so it does not apply.

Search among candidates: generate several partial paths, score them, keep the promising ones and extend. This is the local-search idea from the classical AI module applied to reasoning, and it needs something able to score a partial path, which is the next section.

Verifiers are what make the extra computation pay, and the distinction that matters is what the verifier is.

A real checker — running the tests, executing the code, evaluating the arithmetic, checking the proof — is the strong case. It is exactly right, it cannot be fooled by plausibility, and where one exists this whole approach works extremely well. That is why mathematics and code improved most.

A learned verifier — a model trained to score answers — is weaker and still useful, and it carries the biases the evaluation topic named: length, style, familiarity. Validate it against human judgement before trusting it, exactly as with a judge model.

Process supervision rewards the intermediate steps rather than only the outcome, and the reason it helps is precise: outcome-only reward gives credit to a lucky answer reached by wrong reasoning, so the model learns that wrong reasoning sometimes pays. Rewarding steps removes that. Its cost is that step-level labels are far more expensive to obtain than final answers.

Then the trade this topic exists to force you to make explicitly.

Cost: thinking produces tokens, and output tokens are the expensive kind. A response with a long reasoning passage before it can cost many times a direct answer to the same question. Sampling several attempts multiplies that again.

Latency: more tokens is more time, straightforwardly, and for anything interactive it can move a response from immediate to noticeably slow.

And sometimes worse quality, which is the counter-intuitive part and is worth naming as overthinking. On a simple question, extended reasoning gives more opportunities to talk itself out of a correct first instinct, to elaborate a wrong assumption made early, or to produce a hedged and less usable answer. It is not universal and it is real, and it is why the decision must be measured rather than assumed.

So decide per task class, with measurements, and the classification is not difficult.

Worth it: multi-step problems, mathematics, code that must run, planning, anything where an error is expensive and a delay is acceptable, and anything with a real checker available.

Not worth it: classification, extraction, formatting, summarising, retrieval-grounded question answering, routing — all the high-volume tasks where the work is reading rather than deducing, and where the answer is in the input rather than to be derived.

The measurement is one afternoon: take your evaluation set, run it with the extra computation and without, and record accuracy, latency and cost for both. Then set the routing from the result — cheap and direct by default, escalating to the expensive path when a classifier or a confidence signal says the task warrants it. That is the routing policy from the agent cost topic, applied to reasoning, and it is where the saving is.

Two habits to carry. Bound the thinking, because these systems can produce a very long reasoning passage on a question that did not need one, and a cap protects both cost and latency. And do not show raw reasoning to users unless you have decided to: it is verbose, it sometimes contains discarded wrong turns that read as assertions, and a user who reads a confident wrong intermediate step remembers it. Show the answer; keep the reasoning for your logs, where it is genuinely useful for debugging.

What you should now be able to explain or do

State the difference between spending at training time and at answer time, and why the economics differ. Say what makes trained reasoning different from a prompt technique, and why re-asking for it is redundant. Describe the three test-time mechanisms and what each requires. Distinguish a real checker from a learned verifier and say which is strong. Explain what process supervision fixes and what it costs. Name the three costs including overthinking. Classify task types as worth it or not, measure both paths, and route accordingly. Bound the thinking and keep raw reasoning out of the interface.

Check yourself