The pattern
In the competing consumers pattern multiple worker instances read from the same queue. The broker hands each message to exactly one available worker, so the workers compete for messages and share the load.
Why it scales
- Elastic throughput: add workers to drain a backlog faster, remove them when idle.
- Natural load balancing: a faster or freer worker simply pulls the next message sooner.
- Resilience: if a worker dies mid message, the message becomes visible again for another worker.
What it demands
- Idempotency: redelivery after a crash means a message can be processed more than once, so handlers must tolerate repeats.
- Visibility timeout: a message is hidden while being worked, then reappears if not acknowledged in time.
- Ordering loss: with many parallel workers, strict order across the queue is not preserved.
Key idea
Competing consumers scale throughput by letting many workers pull from one queue, giving elastic load balancing and resilience at the cost of strict ordering and a need for idempotent handlers.