Requirements
- Charge customers and pay merchants exactly once.
- Never lose or double count money, even under retries and failures.
- Keep a full audit trail and reconcile with external providers.
High level design
A payment service records intent, calls a provider, and updates a ledger, with idempotency throughout.
- Idempotency keys: each payment request carries a key so retries do not create duplicate charges.
- Ledger: a double entry record of every movement is the source of truth, written before and after provider calls.
- Provider integration: external gateways are asynchronous, so use webhooks and polling to learn the final state.
Bottlenecks
- Exactly once: networks fail mid call, so combine idempotency keys with a state machine that can resume safely.
- Consistency: money must balance, so use a double entry ledger where debits equal credits and write transactions atomically.
- Reconciliation: provider state can diverge, so a background job compares the ledger against provider records daily.
Treat the ledger as immutable and append corrections rather than editing history.
Key idea
A payment system relies on idempotency keys, an immutable double entry ledger, and reconciliation so every cent is tracked exactly once.