Only ever add to the end
An append only log stores records by writing each new one to the end of a file and never overwriting old bytes. Updates and deletes become new records that supersede earlier ones rather than in place edits.
Why it is fast
- Sequential writes are far faster than random writes on both disks and SSDs.
- There is no need to find free space or update records in place.
- Concurrency is simpler because old data is never mutated.
Reading and finding records
Because records are not sorted by key, a plain log needs an index that maps a key to the byte offset of its latest record. To read a key you look up the offset and seek straight to it. Many storage engines keep this index in memory for speed.
Reclaiming space
A pure append log grows forever, since obsolete records still occupy bytes. Periodic compaction rewrites the log, keeping only the newest version of each key and dropping superseded entries, which bounds the size.
Key idea
An append only log turns every write into a fast sequential append and uses an in memory index plus periodic compaction to make reads quick and to keep the file from growing without bound.