7.5 CI/CD for cloud deployments

What this is and why it exists

A pipeline is the path a change takes from somebody's editor to production, with checks at every step and no way round. Building one changes what a team can safely do more than any other single piece of engineering: releases stop being events, rollbacks stop being feats, and nobody deploys from a laptop at eleven at night. This lesson is the shape of that path, the credential problem at its centre, and the rollback you must have tested.

The vocabulary

• Pipeline — the automated sequence a change passes through: build, test, deploy.
• Stage — one step, which can fail and stop everything after it.
• Gate — the condition that must hold for a stage to proceed.
• Artefact — the built thing a pipeline produces once and promotes onward, unchanged.
• Short-lived credential — access issued for one run and expiring by itself.
• OIDC federation — the pipeline proving its identity to the cloud and receiving temporary credentials, so no key is stored anywhere.
• Blue-green — running two complete environments and switching traffic between them.
• Canary — sending a small share of traffic to the new version first.
• Rolling deployment — replacing instances a few at a time.

The mental model

Build once, promote the same artefact. The pipeline builds one image or package, tests that, and deploys that identical thing to staging and then to production. Rebuilding per environment quietly reintroduces the problem containers were meant to solve, because the thing you tested is no longer the thing you shipped. If the artefact changes between environments, your staging test proved something about a different object.

Each stage exists to make a specific failure cheap. Build fails on code that does not compile. Unit tests fail on logic. Integration tests fail on wiring. Security scanning fails on a vulnerable dependency or a leaked secret. Deployment to staging fails on the environment. Each gate catches its own kind of problem at the point where fixing it is cheapest, and the value of the ordering is that the cheap checks run first. A useful rule: a check that gets skipped when people are busy is a check that does not exist, so it belongs in the pipeline or nowhere.

The tools differ and the stages do not, which is worth knowing before you meet a repository built with a different one. Amazon offers CodePipeline, Azure offers Azure Pipelines, Google offers Cloud Build, and GitHub Actions is widely used against all three. Every one of them expresses the same thing: a sequence of stages, each with a gate, producing an artefact and deploying it. Learn the shape and the tool becomes a syntax question.

Credentials are the part done badly almost everywhere, and there is a right answer. The wrong pattern is a long-lived access key stored in the pipeline's settings: it works until it leaks, it is copied by anyone with access to the configuration, and nothing rotates it. The right pattern is federation — the pipeline presents a token proving which repository and which line of work it is running for, the cloud trusts that issuer, and it returns credentials valid for minutes. No key exists to leak, the trust is scoped to your repository specifically, and the audit log records which run did what. If you improve one thing in an existing pipeline, improve this.

Deployment strategies are three answers to "how do we change what is running without dropping anyone". Rolling replaces instances a few at a time, needs no extra capacity, and briefly runs two versions at once — so the versions must tolerate each other, which mostly means database changes have to be backward compatible. Blue-green runs two complete environments and switches, giving an instant and complete rollback at the cost of running twice the infrastructure during the change. Canary sends a small share of traffic to the new version and watches its error rate before going further, which is the only one that finds a problem your tests did not — and it works only if you can see the new version's metrics separately from the old.

Then rollback, and the hard sentence: a rollback plan you have not executed is a hope. Practise it on a quiet afternoon. Time it. Write down the number, because that number is your real worst case and the one you should quote when somebody asks how bad an incident can get. And know what your rollback does not cover — a schema migration that dropped a column is not undone by redeploying the previous version, which is why database changes are made in expand-and-contract steps: add the new thing, deploy code that writes both, migrate, deploy code that reads the new, and only then remove the old.

Kill switches are the companion to rollback and are worth building before you need one. A feature that can be turned off without a deployment is a feature whose failure is a thirty-second event; the same feature without a switch is a rollback, a rebuild, and a room full of anxious people. Same rule as before: turn it off in staging occasionally, so that the first time it is used in anger is not its first use.

What you should now be able to explain or do

Explain build-once-promote-the-same-artefact and what rebuilding per environment costs. Name the stages of a pipeline and the specific failure each one makes cheap. Describe the credential pattern that stores no key, and say why it is better than a stored one in three respects. Choose between rolling, blue-green and canary for a described situation, and say what each requires of you. Say what a rollback does not undo, and how database changes are shaped to compensate. State the one thing that makes a rollback plan real.

Check yourself