Retries create duplicates
When a request times out, the caller does not know if the work happened. Retrying is the natural fix, but it means delivery is at least once: the operation may run more than once. Without care, a retried payment charges the customer twice.
Idempotency is the answer
An operation is idempotent when applying it many times has the same effect as applying it once. Reading a value is naturally idempotent. Adding one to a counter is not. The standard technique is an idempotency key: the caller attaches a unique key, and the server records which keys it has already processed so a repeat is ignored.
Designing for safe retries
- Make writes idempotent with a dedup key stored alongside the result.
- Add backoff and jitter so retries do not stampede a struggling service.
- Cap the number of retries so a permanent failure does not retry forever.
Key idea
At least once delivery is unavoidable, so make operations idempotent with a dedup key and retry with backoff to stay safe.