The Leader Latch Pattern
Some work must run on exactly one node at a time across a cluster: a scheduled cleanup job, a sequence generator, or a single writer. The leader latch pattern elects one node as leader and lets the rest wait as followers ready to take over.
Election usually rides on a coordination service such as ZooKeeper or etcd. Each contender creates an ephemeral entry; the one with the lowest sequence wins the latch and becomes leader. The ephemeral nature is key: if the leader crashes or its session expires, its entry vanishes and a follower is promoted automatically.
Properties to respect:
- Single active leader Only the holder runs the singleton work.
- Automatic failover A lost session triggers a fresh election with no human intervention.
- Fencing A promoted leader uses a monotonic token so a stalled old leader that wakes up is rejected, preventing two leaders from writing.
The danger is the split brain scenario, where a network partition makes two nodes each believe they are leader. Fencing tokens and a quorum based coordination service guard against acting on stale leadership. The leader latch is the building block behind highly available singletons.
Key idea
A leader latch elects one node for singleton work with automatic failover, and fencing tokens prevent two leaders from acting at once.