The bad record dilemma
A pipeline will eventually meet a record it cannot process, like malformed input or a failed validation. Two naive choices are bad: crashing the whole job blocks every good record behind one bad one, and silently dropping the record loses data with no trace.
The pattern
The dead letter table is a third path. Records that fail processing are written, along with the error reason and original payload, to a separate table or topic. The main pipeline keeps flowing for healthy records, while failures are captured rather than lost.
- No blocking because one poison record does not stop the batch.
- No silent loss because every failure is retained with context.
- Replayable because once the bug or bad data is fixed, dead lettered records can be reprocessed back through the pipeline.
Operating it
Monitor the dead letter table size, since a sudden spike signals an upstream break. Keep enough context, the raw payload and error, to diagnose and replay.
Key idea
The dead letter table routes unprocessable records with their error and payload to a side table, so one bad record neither blocks the pipeline nor is silently lost, and failures can be replayed after a fix.