The write problem
Every active device sends a position ping every few seconds. At scale this is a relentless write heavy firehose. Naively writing each ping straight to a durable database would overwhelm it and waste effort on data that is stale within seconds.
Shaping the stream
- Front with a queue. Pings land in a log or message queue so spikes are buffered and consumers scale independently.
- Keep the latest in memory. A fast key value store holds the current position per device, overwriting on each ping.
- Downsample to durable storage. Only write a reduced trajectory to durable history, not every raw ping.
Backpressure and validation
- Drop or coalesce redundant pings when a client floods.
- Reject pings with impossible jumps or bad timestamps before they pollute the index.
- Stamp each update with a server side time so out of order arrivals can be ordered.
Key idea
Location ingestion buffers a write heavy firehose, keeps only the latest position hot, and downsamples the rest into durable history.