8.2 Alerting that people will actually answer

What this is and why it exists

An alert is an interruption of somebody's life, so the bar for sending one is high and most systems do not meet it. This lesson is about alerts people answer: firing on things users can feel, staying quiet otherwise, arriving with instructions attached, and being a commitment a small team can honestly keep. Get this wrong and the failure is not noisy — it is silent, because everybody has learned to ignore the noise.

The vocabulary

• Symptom — something the user experiences: errors, slowness, a page that does not load.
• Cause — something inside the system: high processor use, a full disk, a restarted process.
• Threshold alert — fires when a number crosses a line.
• Burn rate — how fast you are consuming your allowance of failures, which is a better trigger than a raw threshold.
• Alert fatigue — the state in which alerts are dismissed unread, because most of them have not mattered.
• Runbook — the short document attached to an alert, saying what it means and what to do first.
• Escalation — what happens when the first person does not respond.
• On-call — the arrangement making someone responsible for answering out of hours.

The mental model

Alert on symptoms, not causes. High processor use is not a problem — it is a machine doing work. It becomes a problem when requests get slow or start failing, and that is the thing to alert on, because it is the thing that is actually true when users are suffering. Cause-based alerts have two failure modes and both are common: they fire when nothing is wrong (processor is high, everything is fine) and they stay quiet when something is (processor is low because the service is refusing every request instantly). A short list of symptom alerts — error rate, latency at a high percentile, a critical journey failing a synthetic check, and the freshness of anything with a deadline — covers more real incidents than three dozen cause alerts.

Percentiles rather than averages, for the same reason. An average latency stays comfortable while a tenth of your users wait ten seconds. The ninety-fifth or ninety-ninth percentile is the number that has a person behind it.

Burn rate is the idea that makes thresholds behave. Instead of "alert when errors exceed one percent", express it as how fast the month's allowance of failure is being consumed: burning through a month's budget in an hour is an emergency worth waking somebody for, and burning slowly enough to last three months is not, even if the instantaneous number looks similar. A fast burn pages; a slow burn opens a ticket. That single distinction removes most of the alerts that were technically correct and practically pointless.

Then fatigue, which is the real enemy. Every alert that fires and needs no action teaches the recipient that alerts need no action, and that lesson is learned faster than any procedure. So treat alerts as a maintained set: review what fired each week, delete the ones nobody acted on, tune the ones that fired early, and hold the line that a page must correspond to something a person must do now. If an alert cannot be acted on immediately, it is not a page — it is a ticket, a dashboard, or nothing at all.

Every alert carries a runbook, and it is shorter than people fear: what this means, what the user is experiencing, the first three things to check, and who knows more about this area. Write it when you create the alert, because that is the only moment you understand it fully, and it is what lets somebody who did not build the system be useful at two in the morning.

Finally, honesty about what a small team can promise. Twenty-four-hour cover needs enough people that nobody carries it constantly, and pretending otherwise produces a rota that quietly rests on one person until they leave. A small team can honestly commit to office-hours response, a documented best-effort out of hours, and a very short list of alerts that genuinely justify waking someone. Say which is which, agree it out loud, and design the system to degrade rather than to require heroics — because the alternative is not better reliability, it is the same reliability paid for with somebody's sleep.

What you should now be able to explain or do

Explain symptom-based alerting and give the two ways cause-based alerts fail. Say why a high percentile beats an average. Describe burn rate and say which speed pages and which files a ticket. Name the rule that decides whether something is a page. Write a four-part runbook for an alert you can imagine. State what a five-person team can honestly commit to, and what it should not pretend.

Check yourself