Order is local, not global
In a partitioned log, order is guaranteed only within a partition. There is no global order across partitions because they are written and read independently. This is the central constraint to design around.
Keeping related events ordered
- Partition by key: route all events that must stay ordered, such as all events for one account, to the same partition using a hash of the key.
- Within that partition the events keep their append order, so a consumer sees them in sequence.
Threats to order
- Producer retries: a retried send can reorder records unless idempotent producer settings preserve sequence.
- Parallel consumers: if one partition is processed by multiple threads, output order is lost, so a partition is usually handled by one consumer at a time.
- Repartitioning: changing the partition count rehashes keys, so a key may move and its old and new events can interleave during the change.
Tradeoff with throughput
Strict ordering limits parallelism, since each ordered key stream is effectively serial. More partitions raise throughput but only spread unrelated keys, never reorder a single key.
Key idea
A partitioned log gives order only within a partition, so you partition by key and process each partition single threaded to keep related events in sequence, trading some parallelism for that guarantee.