Why idempotency matters
Most stream systems deliver at least once: a record may be processed more than once after a crash or retry. If your consumer adds to a balance or sends a notification, a duplicate causes real damage. An idempotent consumer produces the same result no matter how many times it sees the same record.
Techniques
- Dedup by event id: keep a store of processed ids and skip a record whose id you have already handled. Record the id in the same transaction as the effect so the check and the write cannot drift.
- Upsert by key: write keyed by a stable id so reapplying overwrites rather than appends, for example setting a row by primary key instead of inserting.
- Conditional updates: apply a change only if a version or state matches, so a stale replay is rejected.
Designing the effect to be safe
- Prefer state set operations such as set balance to one hundred over state mutate operations such as add ten, since set operations are naturally idempotent.
- Where mutation is unavoidable, gate it with a processed id check.
Key idea
Under at least once delivery, an idempotent consumer dedupes by event id or upserts by key so reprocessing the same record never double applies, turning unavoidable duplicates into harmless repeats.