Seeing the Invisible
Background jobs run out of sight, so a stalled pipeline can go unnoticed until users complain. Good observability makes the system legible through metrics, logs, and traces.
The Core Metrics
- Queue depth is the number of pending jobs. Steady growth means workers cannot keep up.
- Oldest job age is how long the front job has waited. This is your real latency signal.
- Throughput is jobs completed per second, the drain rate.
- Failure and retry rate reveals broken handlers or flaky dependencies.
- DLQ depth should sit near zero in a healthy pipeline.
Depth Alone Misleads
A low queue depth can hide a stuck pipeline if nothing is being processed. Always pair depth with age and throughput. Depth near zero with rising age and zero throughput means workers are down, not idle.
Tracing Across Hops
A job often spans a producer, broker, and worker. Propagate a trace id from the originating request into the job so you can follow one unit of work end to end and measure time spent waiting in queue versus running.
Alert on Symptoms
Alert on oldest job age and DLQ growth, which reflect user impact, rather than only on raw depth which fluctuates with normal bursts.
Key idea
Watch queue depth together with oldest job age and throughput, propagate trace ids, and alert on symptoms like job age and DLQ growth.