9.5 Hard budget breakers and runaway protection

What this is and why it exists

Everything else in this module is about spending less. This lesson is about the different problem underneath it: making sure that a mistake of yours cannot produce a bill you are unable to pay. That is not the same as being careful, because the failures here are automatic — they happen while you sleep, they accelerate, and by the time anyone looks the number is already large. For a student learning on a personal account, this is the most important lesson in the module.

The vocabulary

• Runaway — a process that consumes resources without bound because nothing stops it.
• Recursive trigger — a function whose output causes the event that invokes it again.
• Hard cap — a limit that stops work rather than notifying somebody.
• Quota — a ceiling on how much of a resource an account may create.
• Free tier — usage a provider does not charge for, within limits, sometimes only for a period.
• Overage — what you are charged after crossing a free limit.
• Circuit breaker — logic that stops calling something that is failing or costing too much.
• Graceful degradation — reducing what the system does when a limit is reached, rather than stopping entirely.

The mental model

The classic runaways are worth knowing by name, because recognising the shape is most of the defence.

A recursive function: a function reads from a storage bucket, writes its result to the same bucket, and that write triggers the function again. Each generation may spawn more than one, so it grows geometrically and can produce an enormous number of invocations in an hour. It is the most-told cloud horror story because it takes almost nothing to build by accident and needs no traffic at all.

A forgotten accelerator: a GPU machine started for one experiment, left running. Accelerated instances are among the most expensive things you can rent by the hour, and an unattended one bills every hour of every day, weekends included, until somebody notices.

Open egress: a bucket made public, a large file, and traffic you did not expect — a link shared somewhere busy, or an automated crawler. Egress is metered, and unlike compute it has no natural ceiling: the more popular the file, the faster the meter runs.

A retry storm: a failing dependency, a client that retries immediately and forever, and thousands of paid invocations doing nothing but failing. This one is doubly unpleasant because it looks like load, so autoscaling adds capacity to serve it.

The defences arrange themselves in three layers, and the important thing is that only one of them actually stops anything.

Notification is the outer layer: budgets and anomaly alerts, set low, arriving somewhere you read. They are necessary and they are not protection — an alert at three in the morning is a record of what happened, not a limit on it.

Quotas and hard caps are the layer that works, and the difference matters. A quota is a ceiling on what your account may create — how many instances of a type, in a region — and it is what turns an unbounded runaway into a bounded one. Set concurrency limits on functions, so a recursive trigger runs out of permitted concurrency instead of the sky. Set maximum sizes on scaling groups, so a retry storm cannot scale to a thousand machines. Request lower quotas than the defaults where you know your real needs, because a default is sized for a large company and you are not one. On a personal learning account this is the single highest-value hour you will spend.

Automated shutdown is the layer people skip and then wish for: a scheduled action that stops non-production resources every evening, and a rule that terminates anything above a threshold with no owner tag. Combined with the tagging standard from the visibility lesson, this makes forgetting cheap — which is the goal, since forgetting is not a habit anybody has ever successfully given up.

A word on free tiers, because they cause a specific and avoidable shock. Free tiers have two kinds of limit: always-free within a monthly allowance, and free-for-a-period that expires. Crossing either does not stop anything — it starts charging, usually silently, at the ordinary rate. So know which of your resources are on which kind, note the expiry dates somewhere you will see them, and keep a low budget alert running even when everything is supposedly free. The bill that begins the day after a twelve-month period ends is a classic, and it arrives without any action on your part at all.

Finally, degrading when a limit is reached, which is the design-level answer. If a budget is hit, what should the system do? Answering that in advance gives you options better than "stop" and better than "keep spending": disable the expensive feature and keep the cheap ones, serve cached results instead of computing fresh ones, reduce the frequency of a background job, refuse new heavy requests while continuing to serve light ones. Each of those is a decision about what matters most, made calmly in advance rather than at speed by someone looking at a number they did not expect.

What you should now be able to explain or do

Name four classic runaways and describe the shape of each. Say which of the three defensive layers actually stops something, and why the other two are still necessary. Set a concurrency limit and a scaling maximum, and explain which runaway each one bounds. Say why requesting a lower quota than the default is sensible on a personal account. Describe the two kinds of free-tier limit and what happens at each. Give three ways a system can degrade when a budget is reached.

Check yourself