8.11 Serving LLMs: latency, throughput, cost
You can put a per-user cost ceiling on an LLM feature and enforce it.
Before:07. Natural Language ProcessingUnlocks:09. Agentic AI12. Frontier Topics
Serving economics decide whether an LLM feature survives contact with a budget: KV cache memory, continuous batching, paged attention, streaming, timeouts — with vLLM-class servers implementing the hard parts. It sits late in the module because it turns everything into rupees per user. The extrapolation trap is standard: demo costs mislead, and per-user cost at scale with real prompt lengths is the number that decides the feature's fate.
Work through these
KV cache, continuous batching, paged attention
The techniques that make serving efficient: reusing computation across tokens, batching requests as they arrive, and managing memory in pages. These are why a serving stack is faster than a naive loop.
vLLM, TGI and inference servers
Purpose-built inference servers implementing those techniques, which you should use rather than reimplement. Knowing what they do explains their configuration options.
Streaming, timeouts and graceful degradation
Sending output as it is produced, setting time limits, and having something sensible to show when the model is slow or unavailable. Users tolerate slowness far better than silence.
Cost modelling and hard budget breakers
Working out what a feature costs per user and putting an enforced ceiling on it. Without a hard limit, a single loop or a single abusive user can produce a bill nobody expected.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.