10.1 Architecture principles that hold up

What this is and why it exists

Up to here you have learned components. This module is about putting them together for a real problem, and it starts with the part that makes a design reviewable: named principles. Without them, an architecture review is two people's taste in a meeting room. With them, it is a checklist anybody can run, including on their own work, which is where most of the value actually is.

The vocabulary

• Well-architected review — walking a design against a fixed set of pillars and writing down what each one exposes.
• Pillar — one dimension of a review: operational readiness, security, reliability, performance, cost, and sustainability.
• Stateless — a component holding nothing between requests that it would miss if it were restarted.
• Twelve-factor — a set of application conventions that make software cloud-friendly, of which the load-bearing ones are configuration from the environment, stateless processes, and treating logs as a stream.
• Idempotent — an operation that has the same effect whether it runs once or several times.
• Backoff — waiting longer between each retry, rather than retrying immediately.
• Jitter — adding randomness to that wait, so that many clients do not retry in step.
• Timeout — the limit on how long you will wait before giving up on a call.

The mental model

A review is a set of questions asked in a fixed order, and the order matters because it stops you spending the whole hour on whichever pillar you find most interesting. Operational: how do you know it is working, and what happens when it is not? Security: who can reach what, and where do the credentials live? Reliability: what happens when each dependency fails, and where is the data? Performance: what is slow, and is the constraint measured or assumed? Cost: what does this cost per month, and what is the largest line? And sustainability, which after the previous module you will recognise as mostly the cost questions again — utilisation and switched-off resources. One pass, six questions, written down. The output is a short list of the things you now know are weak, and that list is the point; a review producing no findings was not a review.

Stateless design is the principle everything else rests on. A component that keeps nothing important in its own memory or on its own disk can be stopped, replaced, duplicated and moved — which is what makes autoscaling, rolling deployments and recovery from a lost machine possible. Sessions go to a shared store, uploads go to object storage, configuration comes from the environment, and logs are written to the output stream rather than to a file the machine owns. The twelve-factor conventions are mostly this idea applied consistently, and the test is a single question: if this machine vanished mid-request, what is lost that we would miss?

Then the three habits for a distributed system, which is what you now have whether you meant to build one or not.

Timeouts everywhere. A call with no timeout waits forever, and forever is long enough for every worker to end up waiting on the same slow dependency while healthy requests fail behind them. Every network call gets a limit shorter than the patience of whoever is upstream of it.

Retries with backoff and jitter. Retrying immediately turns a struggling service into a failing one, because everybody's retries arrive at once and add load exactly when there is least to spare. Wait longer each time, add randomness so the retries spread out, and cap the number — an unbounded retry is a denial-of-service attack you wrote yourself.

Idempotency, which makes retries safe at all. If a request may be delivered twice, an operation that adds a charge twice is a bug waiting for a bad network day. Give the operation a key the caller supplies, record which keys have been handled, and repeat requests become no-ops. This is the same lesson as the message-buffer topic, arriving from a different direction, and that repetition is a signal rather than a coincidence.

Finally, design for the failure you expect this year rather than the one that makes a better story. For most systems that is: a machine dies, a dependency is slow for ten minutes, a deployment is bad, somebody deletes something, a certificate expires. Those five are ordinary, and a design that handles them well is worth far more than one built around a regional catastrophe it will probably never see. Ask what actually went wrong for you in the last year, and design for that first — it is a better guide than any list of possibilities.

What you should now be able to explain or do

Run a six-pillar review on a design and produce a written list of weaknesses. Say what stateless means operationally and give the one-question test. Explain why every network call needs a timeout, in terms of what happens without one. Describe backoff and jitter and the failure each prevents. Say why idempotency is what makes retrying safe, and how a caller-supplied key implements it. Name the five ordinary failures and design for them before the dramatic ones.

Check yourself