The end to end shape
A real time dashboard needs events to flow from source to screen in seconds. The pipeline is a chain of specialized stages, each tuned for one job.
The stages
- Ingestion lands events in a durable log like a partitioned message queue, which buffers spikes and allows replay.
- Stream processing consumes the log, performs windowed aggregation, and applies sketches like HyperLogLog for unique counts.
- A serving store holds the latest aggregates in a fast read store optimized for the dashboard query pattern.
- The dashboard polls or subscribes to the serving store and renders.
What makes it reliable
- Checkpointing in the processor means a crash resumes without losing or double counting state.
- Idempotent writes to the serving store tolerate replays after failure.
- Backpressure from the log prevents a slow stage from dropping events under load.
The durable log in the middle is what lets each stage scale and fail independently while the dashboard stays fresh.
Key idea
A real time dashboard pipeline chains a durable log, stream aggregation, and a fast serving store, using checkpoints and idempotent writes for reliability.