What Is Pipeline Parallelism? Splitting a Model Into Sequential Stages
A specific form of model parallelism that splits a network into sequential stages across machines, keeping every machine busy at once instead of idle.
Splitting the model into stages, in sequence
Pipeline parallelism splits a model's layers into sequential groups — a "stage" — with each stage assigned to a different machine, so data flows through the machines in order, each one handling its assigned portion of the network before passing its output to the next.
The problem it specifically solves
A naive way to split a model across machines by stage would leave later machines idle while waiting for earlier ones to finish processing the current batch — pipeline parallelism instead overlaps multiple batches in flight simultaneously, so every machine stays busy processing a different batch's stage at any given moment.
How it combines with other strategies
Real large-scale training runs typically combine pipeline parallelism with data parallelism and other forms of model parallelism simultaneously — no single strategy alone is usually sufficient for training the very largest models efficiently.
Frequently Asked Questions
What is pipeline parallelism?
A training strategy that splits a model's layers into sequential stages assigned to different machines, overlapping multiple batches in flight so every machine stays busy rather than idle waiting for earlier stages.
Is pipeline parallelism used alone or combined with other strategies?
It's typically combined with data parallelism and other forms of model parallelism simultaneously — no single distributed training strategy alone is usually sufficient for training the very largest models efficiently.