2.3 Load balancing and autoscaling

What this is and why it exists

One server is a single point of failure and a fixed ceiling. This topic replaces it with a group of interchangeable servers behind a distributor, so a machine can die without an outage and a crowd can arrive without everything falling over. It is also the first place where an architecture decision is visibly a cost decision, because the group grows and shrinks against a meter.

The vocabulary

• Load balancer — the thing traffic reaches first, which hands each request to one of several servers.
• Layer 4 — balancing on addresses and ports, without reading the request; fast and protocol-agnostic.
• Layer 7 — balancing with the HTTP request in view, so it can route by hostname or path and terminate encryption.
• Health check — a repeated probe that decides whether a server is fit to receive traffic.
• Horizontal scaling — adding more machines.
• Vertical scaling — making one machine bigger.
• Scaling policy — the rule that decides when to add or remove machines.
• Cooldown — a pause after a scaling action, so the effect is felt before the next decision.

The mental model

Two mechanisms, and they are separate: the load balancer decides where a request goes, and the scaling rule decides how many servers exist. Confusing them is why people expect a load balancer to fix an overloaded system — it will distribute the overload perfectly and change nothing.

Layer 4 versus layer 7 is a question of how much the distributor reads. Layer 4 sees an address and a port, forwards the connection, and is very fast. Layer 7 reads the HTTP request, which lets it send /api to one group and everything else to another, terminate encryption in one place, and retry a request another server failed to answer. Pay for layer 7 when you need those; do not pay for it when you are moving bytes.

The two mechanisms have a name each in every provider, and knowing all three keeps you from mistaking one vendor's word for the concept. Amazon sells the distributor as Elastic Load Balancing, with the Application Load Balancer as its layer 7 form, and the group as EC2 Auto Scaling. Azure separates them into Load Balancer for layer 4 and Application Gateway for layer 7, with Virtual Machine Scale Sets as the group. Google offers Cloud Load Balancing and Managed Instance Groups. Different words, one idea in each column.

The health check is the quiet centre of the whole design. It is what turns a dead machine into a non-event: probes fail, the balancer stops sending traffic, and users never notice. It is also what turns a slow bug into a catastrophe, because a check that only asks "is the port open" will keep feeding requests to a process that answers instantly with errors — while a check that queries the database on every probe will take the whole group out of service the moment the database has a bad minute. The rule of thumb: check something the application really depends on, keep it cheap, and never make one shared dependency able to fail every server at once.

Horizontal and vertical scaling have different ceilings. Vertical is simple and works on anything, including a database that cannot be split — until you reach the largest machine that exists, and you must stop the machine to resize it. Horizontal has no such ceiling, but only works if the servers are interchangeable, which means no state on the local disk and no assumption that a user comes back to the same machine. Most of the real work of "making it scale" is removing those assumptions, not configuring the group.

Policies come in three shapes. Target tracking names a number to hold — average processor at sixty percent, say — and the system adds and removes machines to keep it there; it is the sane default. Scheduled scaling grows the group before a known event, like results day or a sale, because reacting is always late. Step scaling adds a specified amount when a threshold is crossed, which is for workloads where you know the shape of the response you want. Whichever you choose, machines take minutes to become useful, so every reactive policy is answering the load of a few minutes ago — and a cooldown keeps the system from stacking three responses to one spike.

Then the honest part: scaling out fixes some outages and hides others. It genuinely fixes work that grows with users and can be spread across machines. It hides a slow query, a memory leak, a lock contention problem, or a downstream service with a fixed limit — each of those consumes the extra machines too, and now costs more per hour while failing in the same way. The signal to watch for is a group that grows and never shrinks: that is usually not demand, it is a bug being paid for by the hour.

What you should now be able to explain or do

Explain why a load balancer alone does not make a system handle more load. Choose layer 4 or layer 7 for a given requirement and defend it. Design a health check that catches a broken application without letting one shared dependency remove every server. Say which scaling direction a single relational database can use and why. Pick a policy for a predictable annual traffic peak, and a different one for unpredictable daily traffic. Look at a scaling group that never shrinks and say what you would investigate first.

Check yourself