Durability without copying everything
Full replication protects data by keeping several whole copies, which is simple but expensive. Erasure coding instead splits data into pieces and computes extra parity pieces so the original can be rebuilt even if some pieces are lost, using much less space.
How the math splits data
A scheme is described as k data pieces plus m parity pieces. The original data is divided into k fragments, and m additional parity fragments are computed from them. Any k of the total k plus m fragments are enough to reconstruct the data. The system can therefore lose up to m fragments and still recover.
The overhead comparison
- Three way replication stores three times the data to survive two losses.
- A six plus three scheme survives three losses while storing only one and a half times the data.
So erasure coding gives strong durability at a fraction of the storage cost.
The catch
Reconstruction is not free. Rebuilding a lost fragment requires reading k other fragments and doing computation, which adds network and CPU cost. This makes erasure coding ideal for cold and large data where reads are rare, and replication often better for hot data.
Key idea
Erasure coding splits data into k fragments plus m parity fragments so any k can rebuild the whole, delivering replication grade durability at far lower storage cost but with higher reconstruction overhead.