← Lessons

quiz vs the machine

Platinum1800

System Design

Exactly Once With Idempotency

Combining producer ids and dedup to make each message take effect exactly one time.

6 min read · advanced · beat Platinum to climb

The hard guarantee

Exactly once means each message takes effect one time, no loss and no duplicate. True end to end exactly once is hard because networks retry and processes crash. Practical systems reach it by layering idempotency on top of at least once delivery.

Idempotent producers

A producer tags each message with a producer id and a sequence number. If a network retry resends a message, the broker recognizes the duplicate sequence and discards it. This stops duplicates created on the write path.

Transactional writes

To make the consume process produce cycle atomic, brokers offer transactions: the offset commit and the output writes happen as one atomic unit. Either both land or neither does, so a crash cannot leave the offset committed without the output, or the output without the offset.

Effectively once at the sink

Even so, the final side effect often relies on an idempotency key at the sink, a database upsert or a dedup table keyed by message id, so a replayed message overwrites rather than duplicates.

Flow

Key idea

Exactly once is built, not free: idempotent producers, transactional offset and output writes, and idempotent sinks together make each message take effect once.

Check yourself

Answer to earn rating on the learn ladder.

1. How does an idempotent producer stop duplicate writes from retries?

2. What do broker transactions make atomic for exactly once processing?

3. Why still use an idempotency key at the sink?