Why Sample at All
Recording every trace at scale is expensive in network, storage, and money. Sampling keeps a subset. The question is when to decide.
Head Sampling
Head sampling decides at the start of the trace, before any work happens. The root service flips a coin, sets the sampled flag, and that decision propagates to every downstream span.
- Cheap: drop traces before generating their spans.
- Simple: one decision, no buffering.
- Blind: you decide before knowing if the request was slow or errored.
Tail Sampling
Tail sampling waits until the trace is complete, then decides based on what happened.
- Smart: keep all errors and slow traces, drop boring fast ones.
- Costly: every span must be buffered until the trace finishes.
- Stateful: a collector must hold spans and know when a trace is done.
Many teams combine both: head sample a baseline, tail sample to guarantee errors survive.
Key idea
Head sampling decides cheaply at the start and is blind to outcome, while tail sampling buffers spans to keep errors and slow traces at higher cost.