What Is Data Parallelism? Splitting the Data Across Machines
One of the core strategies for distributed training: give every machine a full copy of the model, but split the training data between them.
Same model, different data, on each machine
Data parallelism gives every machine in a distributed training setup a full copy of the model, but splits the training data so each machine processes a different portion of it simultaneously, computing its own gradients from its own slice of data at the same time.
How the machines stay in sync
After each machine computes its gradients from its own data slice, those gradients get combined (typically averaged) across all machines, and the resulting combined update is applied to every machine's copy of the model identically — keeping all copies synchronized and consistent with each other after every step.
Its limitation: the model still has to fit
Data parallelism requires the entire model to fit in a single machine's memory, since each one holds a complete copy. For models too large for that, model parallelism — splitting the model itself, not just the data — becomes necessary instead, or alongside it.
Frequently Asked Questions
What is data parallelism?
A distributed training strategy where every machine holds a full copy of the model but processes a different portion of the training data simultaneously, with gradients combined across machines after each step.
What is the main limitation of data parallelism?
It requires the entire model to fit in a single machine's memory, since every machine holds a complete copy. For models too large for that, model parallelism is needed instead, or in combination with it.