10.10 Scaling inference
Your service survives a hundred times the traffic.
Scaling inference is standard web engineering applied to models — horizontal replicas, autoscaling, caching, request deduplication — plus model-specific tricks like quantization for latency. Load testing reveals the real ceiling. It sits near the module's end because it stresses everything built. The habit is testing to failure on purpose in daylight, because the alternative is discovering the capacity limit from an outage, at night, with users watching.
Work through these
Horizontal scaling and autoscaling policies
Adding more instances behind a load balancer, with rules deciding when. For prediction services the rules need to account for slow model loading.
Caching layers and request deduplication
Storing previous answers, and recognising when the same request arrives twice. On many workloads caching removes most of the traffic.
Quantization and distillation for latency
Making the model itself smaller and faster, using the compression techniques from earlier modules. This is frequently cheaper than adding hardware.
Load testing and capacity planning
Generating realistic load and measuring what breaks first tells you what capacity you actually have. Estimating it instead is how outages happen at launch.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.