Sync cheaply
Dropbox keeps files identical across devices without re uploading whole files on every edit. The tricks are block level chunking and a sync journal of changes.
Content addressed blocks
A file is split into blocks, and each block is identified by a hash of its contents. If a block already exists in storage, it is not uploaded again. Editing one part of a large file only transfers the changed blocks.
- Files split into fixed size blocks
- Each block keyed by a content hash enables dedup
- Only new or changed blocks are uploaded
The sync journal
A central metadata journal records the ordered list of file changes per account. When a device comes online, it asks for changes since its last known position and applies them, fetching any missing blocks.
Efficiency comes from never moving bytes you already have, and consistency comes from replaying an ordered journal of changes.
Key idea
Chunk files into content addressed blocks so only changed bytes move, and drive convergence from an ordered metadata journal each device replays.