← Lessons

quiz vs the machine

Platinum1750

System Design

Backup And Restore Testing

A backup you have never restored is only a hope, not a recovery plan.

6 min read · advanced · beat Platinum to climb

Backups are not the goal

The goal of a backup is a successful restore. Teams obsess over backup jobs succeeding but rarely verify that the data can actually be brought back. A backup that cannot be restored is worthless, and you usually discover this during a real disaster.

Common ways restores fail

  • Corrupt or incomplete backups that the backup job reported as successful.
  • Missing pieces like the schema, encryption keys, or configuration needed to use the data.
  • Untested procedures so the runbook steps no longer match reality.
  • Too slow: the restore works but takes far longer than the RTO allows.

Test it like you mean it

  • Run periodic restore drills that recover into a clean environment and verify the data.
  • Measure restore time so you know your true RTO, not your hoped for one.
  • Practice point in time recovery to a specific moment, not just the latest snapshot.

Defending against corruption

Replication copies bad data too. If a bug or attacker corrupts records, replicas faithfully replicate the corruption. Immutable or versioned backups that cannot be overwritten let you recover to a known good point before the damage.

Key idea

Only a tested restore proves a backup works, so drill restores regularly and keep immutable copies against corruption.

Check yourself

Answer to earn rating on the learn ladder.

1. Why is an untested backup risky?

2. Why keep immutable backups against corruption?