What stream processing is
Stream processing handles an unbounded sequence of events that never ends. Instead of waiting for all data, the system processes each event or small group as it arrives and updates results continuously.
Core properties
- Low latency is the goal. Results reflect new events within milliseconds to seconds.
- Unbounded input means there is no natural end, so the system must produce partial results over time.
- Continuous operators like maps, filters, and aggregations run forever, keeping running state.
Why it is harder than batch
- The system never sees the whole dataset, so totals are always provisional.
- Events can arrive out of order or late, which complicates time based aggregations.
- State must be kept and recovered safely because the job runs indefinitely.
Stream processing powers dashboards, fraud detection, and alerting where waiting for a nightly run is unacceptable.
Key idea
Stream processing computes continuously over an unbounded event flow to deliver fresh results at the cost of more complex state and ordering handling.