A flattering lie in your numbers
Coordinated omission is a measurement error where a load generator accidentally hides the very latencies it should be catching, producing percentiles that look far better than reality.
How the trap springs
Many testers send a request, wait for the response, then send the next. If the server stalls for one second, the tester also stalls and simply sends fewer requests during the stall.
- The single slow response is recorded.
- But all the requests that should have been sent during that second are never issued, so their would be high latencies never appear.
- The tester and the stall are coordinated, omitting exactly the worst samples.
The result is a percentile chart that understates the tail, sometimes by orders of magnitude.
Measuring honestly
- Send requests on a fixed schedule at the intended rate rather than waiting for the previous reply.
- Or correct afterward by back filling the latencies of requests that the stall delayed, as if they had been sent on time.
These methods restore the missing slow samples so the high percentiles reflect what a real client with steady traffic would feel.
Key idea
Coordinated omission hides the worst latencies because a stalled closed loop tester stops sending, so measure on a fixed schedule or back fill the delayed samples.