10.7 Cloud fundamentals and cost

What this is and why it exists

Three services carry almost every machine learning system: somewhere to compute, somewhere to keep objects, and a managed database. Knowing those three well covers most deployments. What this topic adds is the part people learn expensively — predicting the bill before it arrives, and setting the alert on the first day rather than after the first invoice, because accelerator instances left running over a weekend are a real and repeated story.

The vocabulary

• Compute instance — a machine you rent, by the hour or the second.
• Object storage — durable storage for files, addressed by key rather than path.
• Managed database — a database somebody else operates.
• Interruptible capacity — spare capacity offered at a discount and reclaimable at short notice.
• Serverless — code run per request with no machine to manage.
• Cold start — the delay when a scaled-to-zero service handles its first request.
• Egress — data leaving a provider's network, which is charged.
• Budget alert — a warning when spending crosses a threshold.

The mental model

The three services, and what each is for.

Compute is where training and inference run, rented by time. The important axis is not the provider but the shape: ordinary machines for services and preprocessing, accelerator machines for training and heavy inference, at a completely different rate.

Object storage holds your data, your model artefacts and your outputs. It is cheap, effectively unlimited, durable, and it is the right default for anything large. Two properties shape the design: it is addressed by key rather than by directory, and reading is fast while listing very many keys is not — so lay out keys so you rarely have to enumerate them.

A managed database holds the small structured things: metadata, features, users, predictions worth keeping. Paying somebody to run it buys backups, patching, replication and recovery, which are exactly the things nobody on a small team has time to do correctly.

Accelerator instances are where the money goes, and the choice has two axes. Memory decides what fits — the model, the batch, the activations — and it is the binding constraint far more often than speed, as the scaling topic explained. Speed decides how long you wait. The trap is choosing a large instance because a small one felt slow, when what was needed was mixed precision or a fixed input pipeline: profile before renting more, because the same accelerator idle half the time is the commonest expensive mistake in this field.

Interruptible capacity is a large discount for one condition: it can be taken back at short notice. For training that is entirely workable if you design for it — checkpoint often, resume automatically, treat interruption as normal rather than as failure — and the saving is substantial enough that the engineering pays for itself in one long run. For serving it is much harder, because an interruption is a user's request failing. Design for interruption and the discount is usable; assume it will not happen and it is a hazard.

Serverless, containers and virtual machines trade operational work against control and cost, and the deciding question is how steady the traffic is.

Serverless costs nothing when idle and scales without configuration, so it is right for occasional and unpredictable work — a webhook, a nightly job, a service used a few times an hour. Its costs are cold starts, which for a service that loads a model can be seconds, and per-request pricing that becomes expensive under sustained load.

Containers on a managed platform are the middle and the usual answer: you control the environment, the platform handles placement and scaling, and cost is per running instance. Steady traffic makes this cheaper than per-request, and the environment is the image from the previous topic.

Virtual machines give full control, including the accelerator drivers, and hand you the operating work. They are right for training, for anything with unusual hardware needs, and for cases where a managed platform's constraints do not fit.

The rule: unpredictable and occasional, serverless; steady, containers; specialised or training, machines. And note that cold starts and model loading interact badly — a scaled-to-zero service that loads a large model on its first request is slow in exactly the way users notice.

Then predicting the bill, which is the outcome this topic promises. Cost is a sum of a small number of terms and you can estimate each. Compute: instance rate times hours, and hours means hours it exists, not hours it is busy — an idle instance costs the same as a working one. Storage: volume held, plus the number of operations, which surprises people whose pipeline reads millions of small files. Egress: data leaving the provider's network is charged while data entering is generally not, which is why moving a large dataset out costs more than putting it in. Managed services: their own rates, usually per hour of existence. And the two that appear on the bill and never in the estimate — data transfer between regions, and idle resources nobody remembered.

Write the estimate down before deploying, as a small table of terms, and compare it against the first real bill. The gap is where your mental model was wrong, and looking at it once teaches more than any amount of reading.

Budget alerts on the first day, before the first accelerator instance. The horror stories are real and they share a shape: something was left running, over a weekend or a holiday, and nobody found out until the invoice. Set an alert at a fraction of your expected monthly spend, another at the expected figure, and a hard cap or automatic shutdown where the provider offers one. Alerts go to a person who will act, not to a mailbox nobody reads.

Four habits that prevent the specific failures. Tag every resource with a project and an owner, from the first one, or the bill is an unattributable total. Set automatic shutdown on development machines — an accelerator instance for experimenting should stop itself after idle hours, because everyone forgets one eventually. Set lifecycle rules on storage, moving old artefacts to cheaper tiers and deleting what has no reason to exist. And review the bill monthly, by line, which takes fifteen minutes and is where you find the thing from three months ago that nobody is using.

What you should now be able to explain or do

Name the three core services and say what each is for, including the two properties that shape object-storage design. Choose an accelerator instance by memory and profile before renting more. Design for interruption to use discounted capacity, and say why serving is harder. Choose among serverless, containers and machines by traffic steadiness, and say how cold starts interact with model loading. Estimate a monthly bill term by term, including egress and idle resources. Set budget alerts before the first accelerator instance. Apply the four cost habits.

Check yourself