Why duplicates appear
Under at least once delivery, the broker retries until it sees an ack. If a consumer finishes work but crashes before acking, the message is redelivered and the work could run twice. Deduplication is how we make the second copy a no op.
Two places to dedupe
- Broker side dedup uses a message id or dedup id. The broker remembers ids it has seen inside a time window and refuses to deliver a repeat. This is bounded by the window length.
- Consumer side dedup stores a record of processed ids in a database. Before acting, the consumer checks if the id was already handled, and if so it just acks again.
Practical design
- Give each message a stable id derived from the business event, not a random value, so a resend carries the same id.
- Store processed ids with a TTL so the table does not grow forever.
- Combine dedup with idempotent writes so even a missed dedup check is safe.
Key idea
Deduplication uses a stable message id to detect repeats so at least once delivery produces exactly one effect.