10.4 Serving models
Your model answers an HTTP request in production.
Serving puts a model behind an HTTP endpoint: FastAPI with validation, warm-up, concurrency handling, and a versioned API contract — with batch, real-time and streaming as genuinely different products. It sits at the centre of the production module. The two beginner failures are named: loading the model per request, which multiplies latency by absurd factors, and skipping input validation, which invites garbage in and blames the model for garbage out.
Work through these
FastAPI inference service with validation
A prediction service that validates its input before doing anything with it. Validation at the boundary is what stops malformed requests becoming confusing model errors.
Batch vs. real-time vs. streaming inference
Answering one request at a time, processing a large set at once, or handling a continuous flow. The three have different architectures and different cost profiles.
Model loading, warm-up and concurrency
Loading a model takes time, the first request is slow, and several requests at once compete for memory. Handling all three is what separates a demonstration from a service.
API contracts, versioning and backward compatibility
The shape of the interface, how it changes over time, and keeping old clients working. A model that changes its output shape breaks everything downstream.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.