From source to storage
A metric starts life inside an application as an in memory counter or gauge. The ingestion pipeline is everything between that counter and a queryable store.
Typical stages
- Collection scrapes or receives raw samples from sources.
- Parsing turns wire formats into structured series and labels.
- Validation rejects malformed names or out of range timestamps.
- Buffering absorbs bursts so the store is not overwhelmed.
- Write appends samples to the time series store.
Backpressure and batching
Samples arrive in floods, so the pipeline batches writes and applies backpressure when downstream is slow. A queue between collection and write decouples the two so a slow disk does not stall scraping.
Failure handling
- Drop or buffer when storage is unavailable.
- Deduplicate when sources retry.
- Tag late samples so out of order writes are handled or rejected.
Key idea
Ingestion is a staged pipeline of collect, parse, validate, buffer, and write, with batching and backpressure protecting the store from bursts.