13.6 ML system design interviews

What this is and why it exists

A system design interview asks you to architect something end to end on a whiteboard, and the interviewer is scoring the structure as much as the content. A repeatable framing is the whole game: it stops the conversation wandering, it makes sure you cover what is being assessed, and it prevents the commonest failure, which is diving into model choice in the first two minutes while the requirements are still unknown.

The vocabulary

• Functional requirement — what the system must do.
• Non-functional requirement — how fast, how many, how available, at what cost.
• Ground truth — where the labels come from and how long they take to arrive.
• Candidate generation — narrowing millions of items to hundreds cheaply.
• Ranking — ordering those hundreds carefully.
• Feedback loop — the system's own output shaping its future training data.
• Back-of-envelope estimate — a quick calculation from stated numbers.

The mental model

The framing is five stages in order, and the order is the point.

Requirements — five to ten minutes, and this is where strong candidates spend their opening. What is the system for; who uses it; what decision does the prediction drive; what does an error cost, in each direction; how many requests, at what rate, with what peak; how fast must a response be; does it have to be explainable; what regulation applies. Ask, do not assume, and write the answers where both of you can see them, because everything downstream is judged against them and an interviewer who watches you invent requirements has already learned something.

Data — and this is the stage most candidates skip and interviewers care about most. Where does the training data come from? What is the ground truth, and when does it arrive? — the delayed-label question from the monitoring topic, and it decides the whole architecture. How much is there and how imbalanced? What features are available at prediction time, which is the leakage rule and is the single most common thing to get right in front of an interviewer. Is there a feedback loop where the system's own output shapes what it later learns from?

Model — and only now. Start with the baseline, always: a popularity ranking, a simple rule, a linear model on sensible features. Then say what would make you go further and why. Naming the baseline first is a strong signal; opening with an architecture is the weak move this topic exists to prevent.

Serving — the architecture. Real-time or batch, and defend the choice. Where do features come from at request time, and how do you stop them disagreeing with training. Latency budget across the stages. Caching. Scaling and what happens under a spike. Fallback when the model is unavailable, which candidates forget and which every deployed system has.

Monitoring — how you know it still works. Service metrics and model metrics as separate things. Drift on the inputs. What happens when labels are delayed and what proxy stands in. What triggers a retrain. How you deploy a new model — shadow, then canary — and how you roll back.

Then estimation, which interviewers ask precisely because it separates people who have deployed something from people who have not.

The arithmetic is simple and doing it aloud is the point. From users and actions per user per day you get requests per day; divide by the seconds in a day for the average rate, and multiply by three to five for the peak, because traffic is not uniform. From the request rate and the per-request latency you get the concurrency, and from concurrency and per-instance capacity you get the instance count. From rows per day and bytes per row you get storage per year. And for anything calling a model per request, multiply the per-request cost by the daily volume before anything else, because that number occasionally ends the discussion — which is a good outcome and shows the judgement being assessed.

Round aggressively, say your assumptions aloud, and sanity-check the answer: if it says four thousand servers for a small application, you made an arithmetic error, and noticing that is worth more than the calculation.

Trade-offs and failure modes are the part being assessed, not the diagram.

Make the trades explicit rather than asserting a design: accuracy against latency, freshness against cost, complexity against maintainability, a single strong model against candidate generation and ranking in two stages. Say what you gave up. A candidate who presents an architecture without naming what it costs sounds like somebody who has read about architectures.

And name what breaks first. Which component saturates under load; what happens when a dependency is slow rather than down, which is worse and more common; how a stale feature store shows up; what a feedback loop does over months; what happens on the first day with no data at all. Volunteering failure modes before being asked is the strongest signal available in this format.

Four problems recur across companies and are worth practising until the structure is automatic.

Feed ranking: enormous candidate set, so two stages — cheap generation, careful ranking; implicit feedback; popularity bias and the feedback loop; freshness against relevance; strict latency.

Search: a query intent to satisfy; hybrid keyword and semantic retrieval; reranking; personalisation; the head-and-tail query distribution, where common queries are cacheable and rare ones are where quality is decided.

Fraud: extreme imbalance; delayed and incomplete labels, since you learn about the fraud you caught; an adversary who adapts, which no other problem here has; the cost asymmetry between a missed fraud and a blocked customer; and a hard latency budget inside a payment flow.

Forecasting: time-based validation and rolling-origin backtesting; seasonality and holidays; the classical baselines that must be beaten; per-horizon error; and prediction intervals rather than a point, because a forecast without uncertainty is not usable for a decision.

Practise by talking, not by reading. Set a timer for thirty-five minutes, take one of the four, and speak the whole thing aloud — badly at first. The structure becomes automatic after perhaps six attempts, and once it is, the interview is a conversation about trade-offs rather than an attempt to remember what comes next. That is the entire skill being tested.

What you should now be able to explain or do

Run the five stages in order and spend the opening minutes on requirements rather than models. Ask the data questions, especially where ground truth comes from, when it arrives, and what is available at prediction time. Name a baseline before any architecture. Design serving with feature consistency, latency budget and a fallback. Specify monitoring with service and model metrics, drift, proxies and a deployment path. Estimate rate, concurrency, storage and cost aloud, and sanity-check. State trade-offs and volunteer failure modes. Work the four recurring problems until the structure is automatic.

Check yourself