The problem it solves
You want analytics that are both complete and accurate and available with low latency. Batch jobs are accurate but slow, and streaming is fast but harder to make exact. Lambda architecture runs both and merges them.
The three layers
- Batch layer: stores the immutable master dataset and periodically recomputes complete accurate views over all history.
- Speed layer: processes only recent data in real time to fill the gap since the last batch run, producing approximate up to date views.
- Serving layer: indexes both views so queries merge the accurate batch result with the fresh speed result.
Why two paths
- The batch layer is the source of truth and can correct any errors when it next recomputes.
- The speed layer only needs to cover the short window since the last batch, so its approximations are bounded.
The cost
- Duplicate logic: the same aggregation often must be written twice, once for batch and once for streaming, which drifts and is hard to maintain.
- Operational weight: two processing systems plus a merge layer means more to run and debug.
Key idea
Lambda architecture serves fast and accurate results by pairing a recomputing batch layer with a real time speed layer, paying the price of maintaining the same logic in two places.