What a consumer group is
A consumer group is a set of consumers that cooperatively read a topic. Each partition is assigned to exactly one member of the group, so the partitions are divided up and the group scales horizontally. Different groups read the same topic independently.
When rebalancing happens
The group reassigns partitions, a rebalance, whenever membership changes:
- A consumer joins, for example when you scale out.
- A consumer leaves or crashes and misses its heartbeat.
- The topic gains partitions.
The stop the world cost
In the classic protocol a rebalance is stop the world: every member pauses, revokes its partitions, and waits for a new assignment before resuming. During this pause no progress is made, and frequent rebalances cause noticeable lag.
Reducing rebalance pain
- Cooperative rebalancing moves only the partitions that must change hands rather than revoking everything.
- Static membership gives each consumer a stable id so a brief restart does not trigger a full rebalance.
- Tune timeouts so a slow but alive consumer is not wrongly evicted.
Key idea
A consumer group splits partitions one per member to scale reads, and a rebalance reassigns them on membership change, so minimizing and smoothing rebalances keeps processing steady.