8.2 Tokenizers, context and positional schemes
You can predict a request's token cost before sending it.
Before:07. Natural Language ProcessingUnlocks:09. Agentic AI12. Frontier Topics
Tokens are the currency of LLMs — cost, latency and context limits are all denominated in them — and this topic makes request costs predictable before sending. RoPE scaling and its relatives extend context windows; placement matters because models attend unevenly across long inputs. The working caution is lost-in-the-middle: material buried at the centre of a long context is recalled worst, so where something sits in the prompt is part of prompt design.
Work through these
Counting tokens; cost per 1K tokens
Text is charged and limited by token rather than by word, so counting tokens is how you predict what a request will cost and whether it will fit. The counting tools are part of the standard libraries.
Context window limits and truncation strategies
Every model has a maximum amount of text it can consider, and exceeding it means deciding what to drop. Choosing what to truncate deliberately beats letting a library decide.
RoPE scaling and long-context methods
Techniques for extending how much text a model can handle beyond what it was trained on. They work with caveats, which is why the limits are worth checking rather than assuming.
Lost-in-the-middle and context placement
Models attend less reliably to material in the middle of a long input than at either end. Placing the important material accordingly is a practical and slightly absurd necessity.
Sign in to keep your progress.
Free resources
These videos are on YouTube. Opening the link takes you to YouTube's page. Pressing "Watch here" loads YouTube's player into this page — nothing loads from YouTube until you do. Either way the video comes from Google and uses much more mobile data than a page of text. Something wrong with a link here?
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.