One Upload, Many Outputs
Users upload a single high quality master, but devices need many variants: different resolutions, formats, and bitrates. A transcoding pipeline generates these outputs automatically.
Pipeline Stages
- Ingest receives the upload and validates it
- Queue holds jobs so workers process at their own pace
- Transcode workers produce each variant, often in parallel
- Store writes outputs to object storage
- Publish makes them available through the CDN
Why Asynchronous
Transcoding video is slow and CPU heavy. Doing it inline would block the upload response. Instead the upload returns immediately, a message queue decouples ingest from processing, and a worker pool scales with backlog.
Practical Concerns
- Idempotency so a retried job does not duplicate work
- Progressive availability by publishing a low resolution variant first
- Cost control since transcoding is the most expensive step
Key idea
A transcoding pipeline decouples upload from processing with a queue and worker pool, turning one master into many device variants asynchronously while controlling cost and enabling progressive availability.