The Work Queue And Dispatcher
A common scaling pattern splits a system into two roles connected by a queue. A dispatcher accepts incoming tasks and places them on a shared work queue. A pool of workers pulls tasks from the queue and executes them. This decouples the rate of arrival from the rate of processing.
The decoupling is the point. The dispatcher never blocks on slow work; it just enqueues. Workers consume at their own pace, and you can add or remove workers to tune throughput without touching the dispatcher. The queue absorbs short bursts.
- Decoupled rates Arrival and processing run independently, smoothed by the queue.
- Elastic workers Scale concurrency by changing the worker count, not the dispatch logic.
- Safe handoff The queue must be concurrency safe, since many workers pop while the dispatcher pushes.
Crucially the queue should be bounded so a flood of tasks creates backpressure rather than exhausting memory. With a bound, the dispatcher slows or rejects when workers fall behind. This pattern underlies thread pools, job systems, and message driven services, and it composes with timeouts, graceful shutdown, and rate limiting.
Key idea
A dispatcher enqueues tasks onto a bounded shared queue that workers consume, decoupling arrival from processing and letting you scale concurrency by worker count.