Parallelism on Purpose
Some work is naturally divisible. Process a thousand images, score a million records, or call a service per item. A fan out step splits the parent job into many child tasks that run in parallel. A fan in step waits for all of them and combines the results.
Coordinating the Join
The hard part is knowing when all children finish. Two patterns:
- Counter where a shared count of remaining children is decremented as each finishes. The last one to reach zero triggers the join.
- Completion markers where each child writes a done record, and the join runs when the count of markers equals the expected total.
Store the expected total when you fan out so the join knows the target.
Failure Inside the Group
A child can fail or run twice. Make the counter update idempotent, for example by recording each child id once, so a duplicate completion does not double decrement and trigger the join early.
Partial Results
Decide policy up front. Does the join need every child to succeed, or can it proceed with the successes and flag the failures? Batch jobs often tolerate partial results; financial rollups usually do not.
Key idea
Fan out splits a job into parallel children and fan in joins them, using an idempotent counter against a known total to detect completion safely.