A queue you can share
A distributed queue lets producers on many machines append tasks and consumers on other machines pull them, all in a shared order. It decouples who creates work from who does it, smoothing bursts and allowing consumers to scale independently.
Ordering and visibility
The queue assigns each item a position. Consumers take items in roughly that order. Two design choices shape behavior.
- At least once delivery redelivers an item if a consumer crashes before acknowledging, so work is never lost but may repeat.
- Visibility timeout hides an in flight item from other consumers until it is acknowledged or the timeout expires.
Because consumers can fail mid task, the queue does not delete an item on dequeue. It marks it invisible, and only an explicit acknowledgment removes it.
Avoiding double work
If a consumer is slow, the timeout may expire and a second consumer picks up the same item. This is why consumers should be idempotent, producing the same result if a task runs twice.
Key idea
A distributed queue shares an ordered task list across machines, using visibility timeouts and acknowledgments to survive consumer crashes, which makes idempotent consumers essential.