The Surprise
A user updates their profile, the primary acknowledges, then a failover promotes an asynchronous replica that had not yet received that update. The user reloads and sees the old value. Their own write seems to have vanished. This breaks read your writes consistency, the guarantee that you always see the effects of your own changes.
Why It Happens
Asynchronous replication means the primary commits before replicas confirm. There is always a replication lag window. If a failover promotes a lagging replica, recent writes that lived only on the dead primary are lost or delayed until that node returns.
Mitigations
- Synchronous replica: keep at least one replica fully caught up and only promote it, so no acknowledged write is missing.
- Write fencing tokens: record a position with each write and route the user to a replica that has reached it, falling back to the primary.
- Sticky reads after writes: for a short window send a user's reads to the primary so they always see fresh data.
Key idea
Failover to a lagging asynchronous replica can break read your writes consistency, so promote a synchronized replica or route recent writers to a replica known to hold their write.