10.10 Scaling inference

What this is and why it exists

Scaling a prediction service is mostly ordinary web engineering — more instances, sensible autoscaling, caching — with a few model-specific additions and one behaviour that breaks the usual assumptions: a prediction service takes a long time to become ready. The trap is discovering your capacity from an outage. Test to failure on purpose, before launch, and the number you get is a fact rather than an estimate.

The vocabulary

• Horizontal scaling — adding instances rather than making one bigger.
• Autoscaling policy — the rule deciding when to add or remove instances.
• Cold start — the delay before a new instance can serve.
• Deduplication — recognising that the same request is already in flight.
• Load test — generating realistic traffic to find what breaks first.
• Saturation — the resource that runs out first.
• Headroom — the gap between normal load and the breaking point.
• Capacity plan — what you can serve, with what hardware, at what latency.

The mental model

Horizontal scaling is the default and prediction services have one complication: they are slow to start. A new instance must be placed, the image pulled, the process started, the model loaded and warmed — which can be minutes. Every autoscaling assumption is built for a service that becomes useful in seconds, and this one does not.

So the policy has to lead the load rather than follow it. Scale up on an early signal — request rate or the depth of waiting work — rather than on the lateness that appears once you are already overwhelmed, because by the time latency has risen the new instance is minutes away. Scale down slowly, with a long cooling period, since removing an instance you need back in five minutes costs another cold start. Keep a floor of always-warm instances sized for the load you know arrives. And pre-warm before known peaks — a daily pattern, a campaign, a business cycle — which is a scheduled scale-up and is far cheaper than reacting.

Three things make instances start faster and are worth the effort: a smaller image, as the container topic described; model weights fetched from fast storage or baked into a cached layer rather than pulled slowly over a network; and a readiness check that reports ready only when the model is loaded and warm, so traffic never reaches an instance that cannot serve it.

Caching removes most of the traffic on many workloads, and it is the cheapest scaling there is. Three layers.

Result caching: the same input has the same output, so store it. This works when inputs repeat, which they do far more than people expect — popular items, common queries, repeated documents. Key by the input and the model version, so a new model does not serve stale answers.

Feature caching: the expensive part is frequently fetching or computing features rather than the prediction, and those change more slowly than requests arrive.

Request deduplication: when the same request arrives while an identical one is in flight, wait for the first rather than computing twice. This matters most in exactly the moment it is hardest — a popular item after a cache expires produces a burst of identical requests, and without deduplication every one of them does the full work at once.

Set expiry by how quickly the answer goes stale, not by habit, and measure the hit rate, because a cache with a low hit rate is complexity earning nothing and should be removed.

Making the model smaller is frequently cheaper than adding hardware, and it is the model-specific move. The compression and distillation techniques from the earlier module apply directly: a compressed model uses less memory, so more instances fit on the same hardware and each one loads faster; a distilled small model trained for your task can be a large multiple faster than the model it imitates. Measure quality on your own evaluation set at each step, since the degradation is uneven and the whole point is to know what you traded.

Two ordinary optimisations belong here too. Batching at the server: collecting requests arriving within a few milliseconds and running them together is much more efficient on accelerator hardware, at the cost of a small added latency — a good trade at high volume and a bad one at low. And using the right hardware: many models are faster on ordinary processors at low volume than on an accelerator that sits mostly idle, and that is worth measuring rather than assuming.

Then load testing, which is the point of the topic. Estimating capacity is how outages happen at launch. Generating load and watching what breaks first gives you a number, and the number is usually not where you guessed.

Test with realistic traffic, which means realistic input sizes and a realistic mix — a load test with tiny uniform inputs measures a system you are not running. Ramp until something breaks rather than stopping at your expected load, because the shape of the failure is the information: does latency degrade gradually, or does it collapse; does memory run out; does a downstream dependency fail first; do errors begin at eighty percent of capacity or at a hundred and ten. Find which resource saturates first — accelerator memory, processor, network, the database behind the features — because that is what you scale, and it is frequently not the thing you assumed.

And test the autoscaling itself, not only the steady state: apply a sudden increase and measure how long the system takes to catch up, including the cold start. That number is your true response time to a traffic spike, and it is the one that decides whether an unexpected peak is an inconvenience or an outage.

Write the capacity plan down: requests per second per instance at an acceptable latency, the saturating resource, the cold-start time, the maximum instances your quota allows, and the headroom you keep. Re-run it after any model change, because a new model changes every number in it, and a capacity plan describing the previous model is worse than none — it is confidently wrong.

What you should now be able to explain or do

Say why prediction services complicate autoscaling and write a policy that leads rather than follows. Reduce cold-start time three ways and gate traffic on a real readiness check. Apply result caching, feature caching and deduplication, keyed correctly, and measure the hit rate. Use compression and distillation as a scaling move, measuring quality at each step. Judge server-side batching and hardware choice by volume. Load test with realistic traffic to failure, identify the saturating resource, and test the autoscaling response. Write and re-run a capacity plan.

Check yourself