What an offset is
An offset is the position of a message within a partition. Each consumer tracks the offset of the next message it expects to read. Committing an offset tells the broker how far the consumer has progressed so it can resume there after a restart or rebalance.
When to commit
- Auto commit: the client commits on a timer. Simple but risks committing before processing finishes.
- Manual commit after processing: commit only once work is done, giving stronger delivery guarantees.
The ordering of commit versus process
If you commit before processing and then crash, you skip the message: that is at most once. If you process before committing and crash after processing but before the commit, you reprocess: that is at least once. The choice of order directly sets your delivery semantics.
Where offsets live
Modern brokers store committed offsets in a dedicated internal topic or store, replicated for durability, rather than in the client.
Flow
Key idea
Offsets record consumer progress; whether you commit before or after processing decides at most once versus at least once delivery.