2.5 Resource management, provisioning and scheduling

What this is and why it exists

Somebody decides which of your workloads runs on which machine, and when. On a laptop that somebody is the operating system; in the cloud it is a chain of schedulers, quotas and placement rules that mostly stays invisible until it says no. This topic makes the chain visible, gives you the textbook vocabulary you will meet in examinations, and answers the question the marketing never does: how fast can you really get a hundred more machines?

The vocabulary

• Resource — the five things everything is made of: compute, memory, storage, network, and, increasingly, accelerators.
• Quota (or limit) — the ceiling your account has for a resource in a region, set by the provider and raisable on request.
• Provisioning — creating the resource. Static means fixed in advance, dynamic means created in response to demand, and self-provisioning means the user creates it without asking anyone.
• Over-provisioning — holding more than you need so that you never run short; safe, and paid for by the hour.
• Scheduler — the component that decides which host a workload lands on.
• Bin packing — fitting many differently-shaped workloads onto as few machines as possible.
• Affinity — a rule that two things should be placed together; anti-affinity is the rule that they must be kept apart.
• Elasticity — how quickly capacity can actually follow demand, in both directions.

The mental model

Think of the provider as running a very large game of packing boxes into lorries. Each workload is a box with a shape — so much processor, so much memory, sometimes a demand for a particular kind of disk or an accelerator — and each host is a lorry with a fixed capacity. The scheduler's job is to fit the boxes in without splitting any, and its efficiency is the provider's margin, which is why bin packing is a real engineering concern rather than a puzzle from a textbook.

You influence the packing with two rules and they exist for opposite reasons. Affinity asks for things to be placed close together, because two components that talk constantly are cheaper and faster on the same host or in the same zone. Anti-affinity insists that things be kept apart, and it is the more important of the two: three copies of a service are not redundancy if the scheduler helpfully packed all three onto one host, which then fails. If you take one operational habit from this topic, make it that one — say out loud where your replicas are, and make the platform promise to spread them.

Provisioning has three modes and one trap. Static provisioning fixes capacity in advance, which is predictable and wastes whatever the peak did not use. Dynamic provisioning creates capacity in response to demand, which is what autoscaling does and what elasticity means in practice. Self-provisioning lets any engineer create resources without a gatekeeper, which is the thing that made cloud fast and also the thing that makes bills surprising — the trap is that over-provisioning is the natural individual choice (nobody was ever blamed for a machine being too big) while its cost lands somewhere else entirely. The counterweights are quotas, tagging so every resource has an owner, and a regular look at what is running.

Quotas deserve their own sentence, because they cause a failure people find baffling: a scaling event that stops halfway, not because the provider ran out of machines, but because your account's limit for that instance type in that region was reached. Quotas are per-account, per-region, per-resource, and raising them takes a request and sometimes days. Check them before the event you are scaling for, not during it.

Your textbooks will also give you a family of "as a service" terms beyond the familiar three. Hardware as a service (HaaS) is renting physical machines rather than virtual ones. Licensing as a service (LaaS) is renting the right to use software on a metered basis. Network as a service (NaaS) is renting connectivity and network functions the same way. Inter-cloud resource management is the layer above all of them — arranging for workloads to be placed across more than one provider, which sounds attractive and in practice costs you the specific managed services that made each provider worth using.

Finally, elasticity measured. Ask the question with a stopwatch: from the decision to add a hundred machines, how long until the hundredth is serving traffic? The answer is a sum — the provider's allocation, the boot, the image pull or configuration, the application's own start, and the health check that has to pass before traffic arrives — and it is usually several minutes, not seconds. That number is a design input. If it is longer than your traffic spike, reactive scaling will never catch it and you must keep headroom or schedule ahead.

What you should now be able to explain or do

Name the five resource types and say what a quota is scoped to. Distinguish static, dynamic and self-provisioning, and explain why the last one makes over-provisioning the default individual choice. Explain bin packing from the provider's side. Say what anti-affinity protects against, with a concrete failure. Define HaaS, LaaS and NaaS in a sentence each. Measure your own elasticity end to end and say what design decision the number should change.

Check yourself