The overload problem
If producers publish faster than consumers and storage can absorb, the broker fills up. Without a response, it runs out of memory or disk and crashes. Backpressure is the mechanism that pushes the pressure back toward producers so the system stays stable.
How brokers apply it
- Bounded buffers: queues have a maximum depth; once full the broker stops accepting.
- Blocking or rejecting: a publish call blocks or returns a busy error when limits are hit.
- Throttling: the broker slows ack responses or applies quotas per client to cap rate.
Consumer lag as the signal
Consumer lag, the gap between the latest offset and the consumer's committed offset, measures how far behind consumers are. Growing lag is the early warning that triggers throttling or scaling out consumers before storage fills.
Drop versus block
When you cannot keep up you must choose: block producers to preserve every message but risk stalling upstream, or drop or sample messages to stay responsive at the cost of completeness. The right choice depends on whether the data is critical or sheddable.
Flow
Key idea
Backpressure pushes overload back to producers via bounded buffers and throttling, using consumer lag as the signal, and forces a choice between blocking and shedding.