8.1 Logs, metrics and traces
Describes cloud observability and reliability practice as of August 2026
What this is and why it exists
When something breaks at nine in the evening, you either have data or you have opinions. Observability is the practice of instrumenting a system so that the question "what is happening" has an answer you can look up rather than argue about. Three kinds of data do the work, they answer different questions, and knowing which one to reach for is most of the skill.
The vocabulary
- Log — a timestamped record of one event, with whatever detail you chose to include.
- Metric — a number sampled over time: requests per second, latency, memory in use.
- Trace — the path of one request through several services, with timing at each step.
- Structured logging — logging as fields rather than as a sentence, so it can be queried.
- Correlation ID — an identifier attached to one request and carried through every service and log line it touches.
- Dimension (or label) — an attribute attached to a metric, like the endpoint or the region.
- Cardinality — how many distinct combinations of dimensions a metric has; the thing that decides its cost.
- Span — one step in a trace, with a start, a duration and a parent.
The managed offerings are Amazon's CloudWatch with X-Ray, Azure Monitor with Application Insights, and Google's Cloud Logging and Monitoring with Cloud Trace.
The mental model
Each signal answers a different question, and reaching for the wrong one is why investigations stall.
Metrics answer is something wrong, and since when. They are cheap, they aggregate, and you can keep them for a long time — which is what lets you say "latency doubled at 21:14" and go looking at what happened then. What they cannot tell you is which request, or why.
Logs answer what exactly happened to this one thing. They carry detail metrics cannot, and they cost in proportion to how much you write, so the habit to build is logging events rather than narrating them. Structure is what makes them usable: a line with fields for the request id, the user, the route, the status and the duration can be queried, while the same information in an English sentence can only be searched with a text pattern and hope.
Traces answer where the time went across services. In a system of one process you can mostly do without them; the moment a request crosses three services, a trace is the difference between finding the slow step in a minute and guessing for an hour. It shows the request as a tree of spans, with the one taking nine tenths of the time visible at a glance.
The connective tissue is the correlation ID, and it is worth more than any tool. Generate an identifier at the edge for each request, pass it to every service the request touches, and include it in every log line and every span. That one habit turns three separate piles of data into one story you can follow, and it is the thing that makes a user's complaint — "it failed at about half past two" — a lookup rather than an excavation.
Then cardinality, which is the observability bill's actual driver and takes people by surprise. A metric with dimensions is stored once per distinct combination of those dimensions. Add a dimension with ten values and you have ten times the series; add one with a value per user or per request id and you have created a new metric for every user, forever. The rule: dimensions are for things with a small, bounded set of values — endpoint, status class, region, version. Anything unbounded belongs in a log or a span, where it costs once rather than forever.
Finally, the honest limit. Instrumentation shows you what you thought to record. The failure nobody has instrumented is invisible whatever you have bought, which is why the useful review after every incident is not "why did the tool not tell us" but "what would we have needed to see, and can we record that now".
What you should now be able to explain or do
Say which of the three signals answers each of the three questions, and pick the right one for a described symptom. Explain what structured logging buys over prose. Describe a correlation ID and what it makes possible. Explain cardinality and give one dimension that is safe and one that is ruinous. Say what no amount of observability tooling can show you.
Check yourself
Latency doubled at some point last night. Which signal do you reach for first, and why?
Metrics — they are aggregated and retained, so they tell you that something changed and exactly when. Then logs or traces around that timestamp for why.
A request crosses four services and one of them is slow. What finds it fastest?
A trace. It shows the request as a tree of timed spans, so the step consuming most of the time is visible immediately rather than inferred from four services' logs.
Why log fields rather than sentences?
Because fields can be queried and aggregated — count by status, filter by route, group by version. A sentence can only be pattern-matched, and the pattern breaks the day somebody rewords the message.
You add the user id as a metric dimension. What have you done?
Created a separate time series for every user, permanently. Dimensions must have small bounded sets of values; unbounded identifiers belong in logs or spans, where they cost once instead of forever.
What can no observability platform show you?
Anything nobody instrumented. That is why the question after an incident is what you would have needed to see, and whether you can record it now.
Go deeper
We haven't checked most of these for screen reader use yet.
Back to Logs, metrics and traces: work through the checklist