5.2 Secrets, keys and encryption

What this is and why it exists

A secret is anything that grants access if someone else has it: a database password, an interface token, a private key. This topic is about keeping them out of the places they always end up — repositories, environment dumps, build logs, screenshots in chat — and about what encryption does and does not protect. It ends with the one procedure worth rehearsing before you need it: what to do in the hour a key leaks.

The vocabulary

• Encryption at rest — data stored encrypted on disk, so a stolen disk or a copied backup is unreadable.
• Encryption in transit — data encrypted while it moves, so nothing between the two ends can read it.
• Key management service — the managed store that holds encryption keys and performs operations with them without releasing them. Amazon's KMS, Azure's Key Vault, Google's Cloud KMS.
• Envelope encryption — encrypting data with a data key, and encrypting that data key with a master key held in the key service.
• Key rotation — replacing a key on a schedule so that the amount of data any one key protects is bounded.
• Secret store — the managed place application secrets live: Amazon's Secrets Manager, Azure's Key Vault, Google's Secret Manager.
• Injection at runtime — the application receiving a secret when it starts, from the secret store, rather than carrying it.

The mental model

Start with what the two kinds of encryption actually defend against, because people conflate them and then feel safer than they are. Encryption at rest protects against physical and storage-level theft: a disk taken from a data centre, a backup file copied, a storage bucket read by someone who obtained the file but not your access. It does nothing at all against a caller who has valid credentials — to them the data decrypts, because that is the point. Encryption in transit protects against anything between the two ends: a network someone else operates, a proxy, a compromised link. It does nothing about what happens at either end. Between them they cover the wire and the disk, and neither covers the thing that actually goes wrong most often, which is credentials.

Envelope encryption is the arrangement that makes managed keys practical, and it is worth understanding rather than accepting. Data is encrypted with a data key. The data key is encrypted by a master key that never leaves the key service, and the encrypted data key is stored beside the data. To read, the service is asked to decrypt the data key, and the plaintext data key exists only briefly in memory. Two payoffs follow. You can encrypt terabytes without sending terabytes through the key service. And rotating the master key is cheap, because only the small encrypted data keys need re-encrypting, not the data itself.

Rotation is worth doing for a reason more interesting than compliance: it bounds the damage in time. A key that has never been rotated protects everything you have ever stored, so its exposure is total; a key rotated regularly protects one period's worth. The same logic applies to credentials — and the practical test of whether your rotation works is not whether it is configured but whether it has actually run, unattended, and nothing broke.

Then the daily habit: secrets are injected, never carried. The application does not contain the password, and it does not contain a key that decrypts the password. It starts with an identity — the role from the previous topic — and uses that identity to fetch the secret it needs from the secret store at start-up. Nothing sensitive is in the repository, nothing is in the image, and rotating the secret does not require a rebuild. Environment variables are a reasonable delivery mechanism at the last step, and they are a bad storage mechanism: they show up in process listings, in crash reports, in debug endpoints that print the environment, and in the screenshot someone posts when asking for help.

Finally, the hour a key leaks, which is a procedure and not a panic. Rotate first: create the new credential, put it where things read from, confirm the systems work, then revoke the old one. Rotating first is what actually makes the leaked value harmless, and every other step is investigation that can happen afterwards. Then work out its blast radius — what that credential could reach — and check the audit log for use of it that was not you, especially from unfamiliar addresses or at unusual hours. Then remove it from wherever it leaked, understanding that removing it from a repository's latest commit does not remove it from the history or from anyone's clone; the rotation is the fix, the cleanup is hygiene. And finally write down how it got there, because the same route is still open.

What you should now be able to explain or do

Say precisely what encryption at rest protects against and what it does not, and the same for in transit. Explain envelope encryption and give its two practical payoffs. Say what rotation buys beyond satisfying an auditor. Describe how an application obtains a database password without holding one, and name three places environment variables leak. Recite the leak procedure in order and explain why rotation comes first.

Check yourself