8.3 Decoding and sampling

What this is and why it exists

The model produces a probability for every possible next token. Decoding is the rule that turns those probabilities into one token, and it is the dial between output that is predictable and output that is inventive. Two things make this topic worth care: constrained decoding can force valid structure outright rather than asking for it politely, and the lowest temperature setting is not the reproducibility guarantee everybody assumes it is.

The vocabulary

• Logits — the raw scores over the vocabulary, before they become probabilities.
• Greedy decoding — always taking the highest-scoring token.
• Beam search — keeping several partial sequences and extending the best.
• Temperature — a divisor applied to the scores, flattening or sharpening the distribution.
• Top-k — sampling only from the k highest-scoring tokens.
• Top-p — sampling from the smallest set of tokens whose probabilities sum to p.
• Repetition penalty — reducing the score of tokens already produced.
• Constrained decoding — masking tokens that would break a required structure.

The mental model

Every step produces a full distribution and you choose how to collapse it.

Greedy takes the highest-scoring token every time. It is deterministic given the same computation, and its characteristic failure is being locally sensible and globally dull, because the token that looks best now can commit the sentence to a phrasing with no good continuation.

Beam search keeps several partial sequences and extends each, keeping the best few overall. It finds higher-probability complete sequences and is standard in translation, where there is a right answer. For open-ended generation it is a poor fit, and the reason is instructive: the highest-probability text is bland text, so optimising harder for probability produces something more generic rather than better.

Temperature divides the scores before they become probabilities. Below one, the distribution sharpens and likely tokens dominate; above one, it flattens and unlikely tokens get a real chance. At zero it degenerates to greedy. It is the single most consequential setting and the one people set by folklore — low for extraction, classification, code and anything with a right answer; higher for drafting, brainstorming and variety.

Top-k and top-p restrict what may be sampled at all, and they are guards rather than dials. Top-k keeps the k highest-scoring tokens; its weakness is that k is fixed while the distribution is not, so on a step where the model is confident it admits noise, and on a step where the model is genuinely uncertain it may cut good options. Top-p adapts: keep tokens in descending order until their probabilities sum to p, and the set is small when the model is confident and large when it is not. That adaptivity is why top-p is the better default, and combining a moderate top-p with a chosen temperature covers nearly every case.

Repetition penalties address a characteristic failure of sampling, in which the model falls into a loop and repeats a phrase indefinitely. They reduce the score of tokens already produced, either by a flat penalty or by penalising recent tokens more. They are blunt instruments: penalise too hard and the model cannot use a word it legitimately needs twice, which shows up as strained vocabulary and, in code, as broken output where an identifier must repeat. Tune per use rather than globally, and be especially conservative for code and structured output.

Constrained decoding is the technique that changes what is possible, and it is under-used. Since decoding chooses from a distribution over tokens, you can mask out any token that would make the output violate a grammar, and sample only among the ones that keep it valid. Applied to a schema, this means the output cannot be malformed — not is unlikely to be, cannot be, because there was never a step at which an invalid token was available.

That is a categorically stronger guarantee than asking for a format and validating afterwards, and it removes the retry loop for syntax entirely. Two limits keep it honest: it constrains shape and not truth, so a schema-valid object can still contain a wrong value, and the constraint can occasionally push the model somewhere awkward when the natural continuation was outside the grammar. Where you control the decoder, use it. Where you call a hosted interface, use whatever structured-output mode it offers and validate anyway, which the structured-output topic covers.

Then the point this topic exists to correct: temperature zero is not reproducibility. People set it, get different outputs across runs, and spend a day hunting a bug that is not in their code.

The reasons are in the arithmetic. Floating-point addition is not associative, so summing the same numbers in a different order gives slightly different results. On parallel hardware that order depends on how the work was divided, which depends on the batch — and on a shared service your request is batched with whoever else is being served at that moment. Tiny differences in the scores occasionally flip which token is highest, and one flipped token changes everything after it. Add to that the ordinary facts that a provider may change the model or the serving stack beneath you without a version change you notice, and that mixed precision widens the numerical wobble.

So build the evaluation to tolerate variance instead of assuming it away. Run each case several times and report a distribution rather than a single result. Assert on properties — does it parse, is the schema satisfied, is the fact present, is the forbidden thing absent — rather than on exact strings. Use a fixed seed where the interface offers one, and treat it as reducing variance rather than removing it. And when a difference between two runs matters, check whether it is larger than the run-to-run spread before concluding anything at all.

What you should now be able to explain or do

Explain what each decoding rule does and what greedy's failure is. Say why beam search suits translation and not open-ended generation. Choose a temperature by whether the task has a right answer. Distinguish top-k from top-p and say why adaptivity makes top-p the better default. Use repetition penalties carefully, and say why code needs a lighter hand. Explain what constrained decoding guarantees and what it does not. Give the numerical reasons the lowest temperature still varies, and design an evaluation that tolerates it.

Check yourself