Why idempotency matters
In any at least once system, a consumer will sometimes receive the same message twice. An idempotent consumer is one where processing a message a second time produces the same result as processing it once. This turns scary duplicates into harmless no ops.
The danger of naive consumers
Consider a consumer that adds ten dollars to a balance on each message. A duplicate would add twenty. The operation is not idempotent because its effect depends on how many times it runs.
Patterns for idempotency
- Idempotency keys: each message carries a unique id; the consumer records processed ids and skips repeats.
- Conditional writes: apply a change only if the current state has not already been changed, using a version check.
- Natural idempotency: set the balance to a final value rather than incrementing it.
The processed id store must be durable and checked atomically with the effect, or a crash between the two can still cause a double apply.
Key idea
An idempotent consumer makes repeated delivery safe by ensuring the second processing of a message changes nothing.