10.4 Serving models

What this is and why it exists

A model that answers an HTTP request is a product; a model in a notebook is a result. The distance between them is smaller than people expect and contains two mistakes that account for most bad prediction services: loading the model per request, which makes latency absurd, and skipping input validation, which turns a malformed client request into an unexplained error at two in the morning.

The vocabulary

• Endpoint — one addressable operation of a service.
• Request model — the declared shape of what a caller may send.
• Response model — the declared shape of what the service returns.
• Lifespan — code running once at startup and once at shutdown.
• Warm-up — running a prediction before serving so the first real one is fast.
• Concurrency — several requests in progress at once.
• Contract — the promise the interface makes to its callers.
• Backward compatibility — old callers continuing to work after a change.

The mental model

Load the model once, at startup, not per request. This is the first mistake and it is the most consequential — loading weights from disk takes seconds, and doing it inside a handler means every caller pays that, concurrently, several times over, while memory fills with copies.

The framework has a place for exactly this. Its documentation names the case: "let's imagine that you have some machine learning models that you want to use to handle requests. The same models are shared among requests, so, it's not one model per request, or one per user or something similar", and loading "can take quite some time, because it has to read a lot of data from disk. So you don't want to do it for every request."

The mechanism is the lifespan parameter — "you can define this startup and shutdown logic using the lifespan parameter of the FastAPI app, and a 'context manager'" — written as an async function with a yield, decorated with @asynccontextmanager. Everything before the yield runs at startup; everything after, at shutdown.

And note why that is better than loading at module level, which the documentation also explains: a module-level load would load the model even when you run a small automated test, and the documentation notes that such a test "would have to wait for the model to load". Loading at startup keeps your tests fast, which is the difference between a test suite people run and one they skip.

Warm-up belongs in the same place. The first prediction after a load is slow — memory is allocated, kernels are compiled, caches are cold — and if that first request is a real user's, they wait for it. Run one prediction on a fixed dummy input during startup, and the first real request is fast. It also verifies the model actually loaded and produces the shape you expect, before the service reports itself ready.

Concurrency has one rule worth knowing. Several requests arriving at once compete for the same model and the same memory, and a prediction is compute-bound rather than waiting on anything, so unbounded concurrency does not help — it produces memory pressure and worse latency for everybody. Bound it: a limited number of concurrent predictions, with the rest waiting briefly and then being refused with a clear status rather than accumulating. A service that refuses promptly is far better than one that accepts everything and becomes slow for all of it.

Validation at the boundary is the second mistake, and it is entirely preventable. Declare the request shape as a model with types and constraints and the framework validates before your code runs — the guarantee being the one from the structured-output topic, that after validation "the fields of the resultant model instance will conform to the field types defined on the model". A missing field, a string where a number belongs or a value out of range becomes an immediate, specific, well-formed error naming the field, rather than an exception from inside the model with a stack trace the caller cannot act on.

Declare the response shape too. It documents the contract, it stops an internal field leaking into a public response, and it makes an accidental change to the output shape a test failure rather than a downstream surprise.

The three serving shapes are different products. Real-time answers one request now, and is bound by latency; batch processes a large set on a schedule, is bound by throughput, and is far cheaper per prediction because everything can be batched and no capacity waits idle; streaming handles a continuous flow with bounded delay, and is the most operationally involved. Choose by how quickly the answer is needed — and notice how often the honest answer is "by tomorrow morning", which makes a batch job the right architecture and removes an entire serving problem.

Then the contract, which is what makes a service usable by anybody else. Version the interface explicitly in the path, and treat the shape as a promise. Adding an optional field is compatible; removing a field, renaming one, changing a type, narrowing an accepted range or changing the meaning of a value is not — and changing the meaning is the one that hurts, because nothing fails and everything downstream is quietly wrong.

Two practices that keep it manageable. Separate the model version from the interface version, and return the model version in every response: a retrained model behind an unchanged interface is an ordinary event, and the caller can still see which model answered — which is the lineage the versioning topic asked for. And run versions side by side during a migration rather than switching, so callers move at their own pace and rollback is a routing change.

Two more things a real service needs. A health endpoint distinguishing "the process is up" from "the model is loaded and ready", because a load balancer sending traffic to a process still loading its model produces a burst of failures at every deployment. And structured logs with a request identifier, including the model version, the input shape, the latency and the outcome — never the raw input, which may contain personal data, as the responsible-AI topic requires.

What you should now be able to explain or do

Load a model once at startup using the lifespan mechanism, and say why module level is worse. Add a warm-up prediction and say what it verifies. Bound concurrency and refuse promptly. Validate requests and responses at the boundary with declared models. Choose among real-time, batch and streaming by when the answer is needed. Version the interface, name which changes break compatibility, and say which one hurts most. Separate model version from interface version and return it. Add a readiness health check and structured logs without raw input.

Check yourself