Data Parallelism vs Task Parallelism
When you want a program to run faster on many cores, you must decide how to cut the work into pieces. There are two classic shapes, and most real programs blend them.
Data parallelism applies the same operation to many pieces of data at once. If you must add one to every element of a large array, you split the array into chunks and give each core a chunk. Every worker runs identical code on different data. This scales naturally because adding more data usually means adding more independent chunks.
Task parallelism splits the work by the kind of operation. One core might decode an image while another compresses audio and a third writes to disk. Each worker runs different code, and the speedup is limited by the number of distinct tasks you have.
- Data parallel Same instructions, many data slices, scales with input size.
- Task parallel Different instructions, runs in parallel, scales with the number of stages.
- Blend A pipeline of tasks where one stage is itself data parallel is very common.
The choice shapes how you balance load. Data parallel work divides evenly when chunks are equal, while task parallel work can stall if one task is far heavier than the rest.
Key idea
Data parallelism runs the same operation across many data slices, while task parallelism runs different operations at once, and most fast programs combine both.