8.2 Tokenizers, context and positional schemes

What this is and why it exists

Tokens are the currency. Cost is counted in them, the length limit is counted in them, and latency scales with them, so a habit of estimating token count before sending a request is the difference between a feature with predictable economics and one that produces a surprise. This topic is that habit, what happens when a request will not fit, and the uncomfortable finding that material in the middle of a long input is attended to less reliably than material at either end.

The vocabulary

• Token — one unit of model input or output, usually a word or part of one.
• Context window — the maximum number of tokens a request may occupy.
• Prompt tokens and completion tokens — what you send, and what comes back.
• Truncation — dropping material so a request fits.
• Positional scheme — how the model is told where each token sits.
• Relative position — encoding separation between tokens rather than absolute places.
• Position interpolation — compressing position values so trained positions cover a longer input.
• Effective context — how much of the window the model actually uses well.

The mental model

Count before you send. Every provider and every open-weight model ships a tokenizer, and running your text through it gives an exact count in milliseconds. From there the arithmetic is yours: the published rate per thousand tokens, multiplied by your prompt length plus your expected output length, multiplied by requests per user per day. Do that once, on your real prompts rather than a short example, and the answer frequently changes the design — a system prompt of two thousand tokens sent on every request is a fixed cost per request forever, and moving half of it into retrieved context that only appears when relevant can halve a bill.

Three counting facts worth knowing. Output tokens usually cost more than input tokens, so an instruction to be concise has direct economic effect. The tokenizer is model-specific, so a count from one is an estimate for another rather than a number. And the split is uneven across languages: text in scripts under-represented in a vocabulary's training data takes considerably more tokens for the same meaning, so the same content costs more and consumes more of the window for some of your users than for others. Measure that on your own data rather than assuming, because it is the kind of unfairness that is invisible until somebody looks.

When a request will not fit, decide what to drop rather than letting a library decide. The default behaviour of many wrappers is to cut from one end, which will happily remove your system instructions or the most recent turn of a conversation, and the failure looks like the model ignoring you.

Four strategies, chosen by what the material is. Keep the ends: system instructions and the latest turn are almost always the things you cannot lose. Summarise the middle: replace older conversation with a compact running summary, which costs one extra request and preserves continuity. Retrieve rather than include: instead of sending a whole document, send the parts a search says are relevant, which is the retrieval topic and is usually the right answer at any real scale. Split the task: process a long document in pieces and combine the results, which is slower and bounded.

Extending the window is a real technique with real caveats. The positional schemes from the transformer topic decide what is possible: absolute learned positions have nothing at all for a position beyond the longest seen in training, while the relative schemes encode separation and therefore degrade rather than break. The rotary scheme in particular can be extended by compressing the position values so that the range the model was trained on covers a longer input, sometimes with a short fine-tune to adapt. It works. What it does not do is make the model as good over the extended range as it was over the trained one, so a stated maximum is a capacity rather than a promise, and the honest way to know is to test recall at your own lengths with your own material.

Which brings us to the finding that shapes how you write a long prompt. Models attend less reliably to material in the middle of a long input than to material near the beginning or the end. Put a critical fact at position one, or at the end immediately before the question, and it is used; bury the same fact in the middle of many pages and recall drops measurably. This is well documented, it is somewhat absurd, and it is a property you have to design around rather than argue with.

So the practical placement rules. Put instructions at the start, where they anchor everything. Put the question, and anything the answer depends on most, at the end, closest to where generation begins. Order retrieved material by relevance rather than by whatever the search returned, so the strongest evidence sits at an edge. Keep the total short even when a longer one would fit, because effective context is shorter than stated context and a smaller, better-ordered prompt routinely beats a larger complete one. And when a long-context system underperforms, test it: put a known fact at the start, the middle and the end of a realistic input and ask for it back. That takes twenty minutes and tells you what your actual working length is.

One closing note on economics. A longer prompt costs on every request, forever, while the engineering to shorten it is paid once. Reducing a prompt by a third is usually worth more than a great deal of the tuning people spend time on, and it improves latency and recall at the same time.

What you should now be able to explain or do

Count tokens for a real prompt and compute a per-user cost from a published rate. Say why output length matters economically and why counts are model-specific. Recognise the uneven cost across languages and check it on your own data. Choose a truncation strategy deliberately and say what the default gets wrong. Explain what extending a context window does and does not buy, and why the schemes differ. State the middle-of-the-input effect and place instructions, evidence and the question accordingly. Test your own effective context.

Check yourself