Compressing timestamps
Metrics are usually scraped at a fixed interval, so timestamps rise by nearly the same amount each time. Delta of delta encoding exploits this regularity to store timestamps in very few bits.
Three layers of difference
- The raw timestamp is large.
- The delta is the gap from the previous timestamp, which is roughly the scrape interval.
- The delta of delta is the change in that gap from one sample to the next, which is usually zero.
When scraping is perfectly regular, every delta of delta is zero and can be stored in a single bit.
Variable length codes
When the gap shifts slightly, such as a jitter of a few milliseconds, the small nonzero delta of delta is stored using a short variable length code, with longer codes reserved for rare large jumps.
Why it pairs with Gorilla
Gorilla compresses values while delta of delta compresses timestamps, so together a sample of timestamp and float often costs only a handful of bits.
Key idea
Delta of delta encodes the change in the gap between timestamps, which is zero for regular scrapes, shrinking timestamp storage to about one bit per sample.