Requirements
- Accept image uploads and produce resized and reformatted variants.
- Process asynchronously without blocking the upload response.
- Serve variants quickly worldwide.
High level design
Uploads land in storage, then a queue drives workers that generate variants.
- Upload: the original is stored in object storage and an event is enqueued.
- Workers: pull jobs, generate thumbnails and formats, and write variants back to storage.
- Delivery: variants are served through a CDN, optionally generated on first request and then cached.
Bottlenecks
- Bursty uploads: a queue smooths spikes and lets workers scale independently.
- Redundant work: deduplicate by content hash so identical uploads are processed once.
- On demand cost: generating every variant up front wastes storage, so derive rare sizes lazily on first request.
Tradeoffs
- Eager variant generation is fast to serve but wastes storage on unused sizes.
- Lazy on first request saves storage but adds latency to the first viewer.
Key idea
An image pipeline decouples upload from processing with a queue and workers, then serves CDN cached variants, choosing eager or lazy generation per size.