8.11 Serving LLMs: latency, throughput, cost

What this is and why it exists

Serving economics decide whether a feature survives contact with real usage. A demonstration costs nothing because three people used it twice; the same feature with real prompt lengths and real traffic has a per-user cost that either works or does not. This topic is what makes serving efficient, what to use rather than reimplement, and the one control that separates a feature with a budget from a feature with a surprise bill: an enforced ceiling.

The vocabulary

• Prefill — processing the prompt, in parallel, before generation starts.
• Decode — producing output tokens one at a time.
• KV cache — stored keys and values for earlier positions, reused each step.
• Static batching — waiting to collect a batch, then running it to completion.
• Continuous batching — adding and removing requests from a running batch.
• Paged attention — managing cache memory in fixed blocks rather than contiguously.
• Time to first token — how long before output starts appearing.
• Budget breaker — an enforced limit that stops spending.

The mental model

Generation has two phases with completely different costs, and every serving decision follows from that. Prefill processes the whole prompt at once, in parallel, so it is compute-bound and fast per token. Decode produces one token at a time, each needing the whole model's weights read from memory to produce a single token, so it is memory-bandwidth-bound and slow per token. A long prompt with a short answer and a short prompt with a long answer cost very differently, and the second is usually the expensive one.

The cache is what makes decoding viable. As the decoder topic explained, the keys and values for earlier positions do not change, so they are stored and reused instead of recomputed, and work per step becomes constant rather than growing with the sequence. The cost moves to memory, and that memory is the binding constraint on how many conversations a server can hold at once. A serving stack is largely a set of answers to "how do we fit more cache".

Continuous batching is the first answer, and it is a large win. Batching many requests through the model together is what makes an accelerator efficient at all. The naive approach collects a batch, runs it until every request finishes, then starts the next — which means every request in the batch waits for the longest one, and a short answer sits idle behind a long one. Continuous batching instead removes a request from the batch the moment it finishes and admits a waiting one in its place, so the batch stays full and nobody waits on somebody else's length. Throughput improves by a large factor, and it is entirely a scheduling change.

Paged attention is the second answer, and it is a memory-management idea borrowed from operating systems. Allocating each request's cache as one contiguous block requires reserving space for the longest output it might produce, and most requests do not produce it, so a great deal of reserved memory sits unused while new requests are refused for lack of space. Managing the cache in fixed-size blocks instead — allocated as needed, not necessarily adjacent — removes that waste and lets blocks be shared between requests with a common prefix, which is exactly the case when many requests share one long system prompt. The purpose-built servers describe this as "efficient management of attention key and value memory", alongside "continuous batching of incoming requests, chunked prefill, prefix caching".

Use a purpose-built inference server rather than reimplementing this. The techniques above are substantial engineering, they interact, and the difference in throughput between a naive loop and a proper server is not a percentage. Knowing what they do is what makes their configuration legible — why a maximum sequence length setting exists, why memory utilisation is a tunable fraction, why prefix caching helps you enormously if all your requests share a system prompt and not at all if they do not.

Streaming, timeouts and degradation are the user-facing half, and they matter more than the numbers suggest. Stream the output as it is produced: total time is unchanged and perceived responsiveness is transformed, because time to first token is what a person experiences as speed. Set timeouts at every layer, and make them shorter than the layer above so a failure surfaces where you can handle it rather than as a browser hanging. And decide in advance what to show when the model is slow or unavailable — a cached answer, a simpler non-model path, a clear message, a retry with a smaller request. Users tolerate slowness far better than silence, and far better than a spinner that never resolves.

Then the part this topic exists for: cost modelling with an enforced ceiling.

Model the cost per user, not per request, and do it with real numbers. Take your actual prompt length including the system prompt and any retrieved material, your actual output length, and the real distribution of requests per user per day — which has a long tail, because a small number of users will do far more than the median. Multiply by the published rate for a hosted model, or by hardware and utilisation for a self-hosted one. Do this before building, because the answer sometimes says the feature does not work at the price you can charge, and that is much cheaper to learn now.

And then enforce a ceiling, because a model is arithmetic and an incident is not. Without a hard limit, three things produce a bill nobody expected: a loop where a system calls itself or retries without limit; one user, curious or abusive, sending requests as fast as they can; and a change in prompt length that multiplies every request's cost quietly.

The controls that actually work, in order of how much they save. A hard per-user cap on requests and tokens per period, enforced in your code before the request goes out, that returns a clear message rather than degrading silently. A maximum output length on every single call, because the default is generous and an unbounded generation is an unbounded cost. A maximum input length, checked before sending, so an enormous pasted document is refused rather than billed. A loop budget: any process that can call the model more than once carries a maximum number of calls and a maximum total spend, checked every iteration. And provider-side spending limits with alerts, as the last line rather than the first, because they tell you after the money is gone.

Log token counts per request from the first day, with the user and the feature attached. Without that you cannot answer where the money went, and the question always arrives after the bill rather than before.

What you should now be able to explain or do

Distinguish prefill from decode and say which is memory-bound. Explain what the cache saves and where the cost moves. Say what continuous batching changes and why it is a large win. Explain paged attention as a memory-management idea and say when prefix sharing helps. Argue for a purpose-built server and read its configuration knowledgeably. Use streaming, layered timeouts and a defined degradation path. Model per-user cost with real prompt lengths and a long-tail request distribution. Put five enforced controls in place and log token counts per request.

Check yourself