3.5 Queues, streams and analytics stores

What this is and why it exists

Two problems, one topic. The first is coupling: when service A calls service B directly, B being slow makes A slow and B being down makes A fail. The second is analysis: running reports against the database that serves your users is how a Monday-morning dashboard takes the site down. Both are solved by putting something in between — a buffer for messages, and a separate store built for reading enormous amounts at once.

The vocabulary

• Message queue — a buffer one system writes to and another reads from, so neither has to be available at the same moment. Amazon has SQS, Azure has Service Bus, Google has Pub/Sub.
• Producer and consumer — the system putting messages in, and the system taking them out.
• At-least-once delivery — the guarantee you almost always get: every message arrives, and some arrive more than once.
• Idempotent — an operation that has the same effect whether it runs once or five times.
• Stream — an ordered, replayable log of events that many readers can consume independently: Amazon's Kinesis, Azure's Event Hubs, Google's Pub/Sub.
• Batch ingestion — collecting data and loading it on a schedule.
• Data warehouse — a store designed for large analytical reads: Amazon's Redshift and Athena, Azure's Synapse and Fabric, Google's BigQuery.
• Columnar storage — storing each column together rather than each row, so a query touching three columns reads only those.
• Poison message — one that fails every time it is processed and would otherwise be retried forever.

The mental model

Start with what the buffer changes. Without it, a slow or absent consumer is the producer's problem immediately. With it, the producer writes and moves on, work accumulates safely, and the consumer catches up at its own pace — which converts an outage into a delay. That is the whole value, and it is large. The cost is that everything becomes asynchronous: the user no longer gets the result in the same request, so you need somewhere to report progress, and errors surface somewhere other than where they started.

Then the guarantee, which is the thing most people get wrong. Almost every such system promises at-least-once delivery, and that promise is exactly as sharp as it sounds: duplicates are normal, not exceptional. They arise from ordinary events — a consumer that processed a message and died before acknowledging it, a network timeout, a redelivery after a visibility timeout expired. So the consumer must be built to survive seeing the same message twice, and there is only one reliable way: make the operation idempotent. Give each message an identifier and record which ones have been handled; or design the effect so repetition is harmless, like setting a value rather than incrementing one. "Exactly once" as a system property is either much more expensive or much narrower than it sounds, and idempotent consumers are what real systems rely on instead.

Streaming and batch are two shapes of movement, not two levels of sophistication. Streaming carries events continuously and is what you want when the value of data decays quickly — a fraud check, a live dashboard, an alert. Batch collects and loads on a schedule, and it is simpler, cheaper, easier to reprocess when the logic was wrong, and entirely correct for anything reported daily. A stream is also a log rather than a buffer: readers hold their own position and can rewind, which is what lets you add a second consumer later and have it read the whole history.

Analytics stores are the other half. A row-oriented database stores each record together, which is ideal for "fetch this one order" and wasteful for "average this one column across two hundred million orders", because it must read every row entire. Columnar storage keeps each column together, so that second query reads one column and skips the rest — the difference is often one or two orders of magnitude, and it compresses far better because a column holds values of one kind. That is why analytics belongs in a warehouse rather than in the production database, and why the answer to a slow report is usually to move the query rather than to add an index.

Finally, poison messages, and the arrangement that contains them. A message that fails every attempt will be retried forever, consuming your workers and hiding every other failure behind its noise. So a limit is set on attempts, and messages that exceed it are moved aside into a separate holding place — a dead-letter queue — where nothing retries them and a person can look. Two rules make that arrangement worth having rather than decorative: alert when something lands there, because a silent one is a message you have lost with extra steps; and make sure a message can be put back after the bug is fixed, because the point is to defer the failure, not to bin it.

What you should now be able to explain or do

Explain what a buffer between two services changes, and what it costs. State the delivery guarantee you should assume and two ways to build a consumer that survives it. Choose streaming or batch for three different needs and defend each. Explain why a columnar store answers an analytical query so much faster. Describe the dead-letter arrangement including the two rules that make it useful.

Check yourself