When supply outruns demand
In a pipeline, a producer may emit records faster than a consumer can handle. Without control the consumer's buffers fill, memory grows, and the process may crash.
What backpressure does
Backpressure is a signal that flows upstream telling a fast stage to slow down. It matches the producer's rate to the slowest consumer so the pipeline stays stable.
How it is applied
- Pull based systems like Kafka let consumers fetch at their own pace, so a slow consumer simply requests less. The durable log absorbs the lag.
- Bounded buffers block or pause a producer when the buffer is full.
- Credit based flow control gives a sender a budget of records it may send before the receiver grants more.
When you cannot slow the source
If the source cannot be slowed, like live sensor data, you must choose a load shedding policy: drop records, sample, or batch. The choice is a deliberate tradeoff against accuracy.
Key idea
Backpressure pushes a slow down signal upstream to match producers to the slowest consumer, and when the source cannot slow you must shed load deliberately.