3.5 Queues, streams and analytics stores
You can decouple two systems and analyse data without hammering your production database.
Before:02. ComputeUnlocks:04. Networking in the Cloud08. Observability & Reliability
The same thing, three names
One idea, sold under a different brand by each provider. Learn it once and you can read any job advertisement.
- On AWS
- SQS/Kinesis + Redshift/Athena
- On Azure
- Service Bus/Event Hubs + Synapse/Fabric
- On Google Cloud
- Pub/Sub + BigQuery
Two systems that call each other directly fail together, and this topic covers the decoupling machinery: message queues with their at-least-once reality, streaming against batch ingestion, and warehouses with columnar storage for analysis that stays off the production database. It closes the storage module because it composes everything earlier into data that moves. The idea that repays the most attention is at-least-once delivery — messages can arrive twice, so handlers must be idempotent, and the dead-letter pattern exists for the messages that keep failing.
Work through these
Message queues and the at-least-once reality
Putting a message queue between two systems lets each fail without taking the other down, and the delivery promise is usually at least once rather than exactly once. Designing for a message arriving twice is therefore ordinary, not defensive.
Streaming vs batch ingestion
Streaming processes events as they arrive, batch processes them in scheduled groups, and the choice changes both the cost and how fresh the answers are. Many systems sensibly do both.
Data warehouses and columnar storage
A data warehouse stores data by column rather than by row, which is why it can scan enormous tables for a report quickly. Sending analytics there instead of at your production database is the point.
Dead-letter queues and poison messages
A message that keeps failing has to go somewhere it can be inspected rather than retried forever, and dead-letter queues are that somewhere. Without one, a single bad message can stall a pipeline indefinitely.
Sign in to keep your progress.
Free resources
We haven't checked most of these for screen reader use yet.
Links last checked 29 Aug 2026.
Stuck here?
Ask a mentor. A real person answers, and they can see exactly which topic you're on. Usually within a couple of working days.
Checking your session…
Topics shown in module order.