Two timing models
Data processing splits along when work happens relative to when data arrives.
- Batch processing collects records over a window, then processes the whole group on a schedule, such as hourly or nightly. It is simple, easy to retry, and efficient for large volumes.
- Stream processing handles each record or tiny micro batch within seconds of arrival. It powers dashboards, fraud checks, and alerts that cannot wait for the next batch.
Trade offs
Batch jobs are cheaper per record and easier to reason about because the input set is fixed when the job starts. Streams give low latency but must handle unbounded input, out of order events, and state that lives across time. Many teams run both in a layered design where a stream gives fast approximate answers and a batch job later produces the exact corrected result.
Key idea
Batch trades latency for simplicity over a fixed input, while streams trade simplicity for low latency over unbounded input.