← Lessons

quiz vs the machine

Platinum1780

System Design

Exactly Once Streaming

Achieving effectively exactly once results despite retries by combining idempotency and atomic commits.

6 min read · advanced · beat Platinum to climb

What the guarantee means

Failures force retries, and retries risk processing an event twice. Exactly once means each event affects the result as if processed a single time, even though the system may physically replay it. It is really effectively once for the output state.

Why at least once is not enough

At least once guarantees no event is lost but allows duplicates. A duplicated payment or double counted metric is often unacceptable, so we need stronger semantics for state and output.

How it is achieved

  • Idempotent writes make a repeated update harmless, for example keying by event id so a replay overwrites rather than adds.
  • Transactional sinks commit output and the consumed offsets together atomically, so a failure rolls back both.
  • Snapshot aligned commits tie state checkpoints to output commits so recovery restores a consistent pair.

The boundary

Exactly once is easiest when the system controls both source offsets and sink commits. Talking to an external system that lacks transactions usually reduces the guarantee to idempotency only.

Key idea

Exactly once means each event affects state once despite replays, achieved by pairing idempotent writes with atomic commits of state and output offsets.

Check yourself

Answer to earn rating on the learn ladder.

1. What does exactly once really guarantee?

2. Why is at least once often insufficient?

3. Which pairing supports exactly once output?