Offsets mark progress
A consumer tracks how far it has read in each partition with an offset. Committing an offset records that progress so a restarted consumer resumes from the right place.
The order that defines semantics
The risk comes from the order of two steps: processing a record and committing its offset.
- Commit after processing gives at least once delivery. If the consumer crashes after processing but before committing, it reprocesses the record on restart. No loss, possible duplicates.
- Commit before processing gives at most once delivery. If it crashes after committing but before finishing, the record is skipped. No duplicates, possible loss.
Auto commit caution
Kafka can auto commit offsets on a timer. This is convenient but commits may land before processing finishes, quietly causing loss or duplicates. Many teams disable it and commit manually after work succeeds.
Designing for at least once
Because at least once is the safe default, make your processing idempotent so a replayed record causes no harm.
Key idea
Committing an offset after processing gives at least once with possible duplicates, while committing before risks loss, so prefer manual commits plus idempotent handling.