What Is Sparse Training? Not Every Parameter Needs to Activate Every Time
A training approach where only a subset of a model's parameters are active for any given input, aiming to cut compute cost without sacrificing overall capacity.
Only a subset of the model active at once
Sparse training designs and trains a model so that only a subset of its total parameters are actually active — and cost compute — for any given input, rather than every single parameter participating in every single computation the way a fully "dense" model works.
Why sparsity is an appealing tradeoff
A model can have a very large total parameter count — and the representational capacity that implies — while keeping the actual compute cost per request closer to that of a much smaller dense model, since only a fraction of the full parameter set gets used for any individual input.
The clearest real-world example
Mixture of experts is the most widely deployed form of sparse training in modern LLMs — activating only a subset of specialized "expert" sub-networks per input, rather than routing every input through the model's entire parameter set.
Frequently Asked Questions
What is sparse training?
A training approach where only a subset of a model's total parameters are active for any given input, rather than every parameter participating in every computation as in a fully dense model.
What is the main benefit of sparse training?
A model can have a very large total parameter count and representational capacity while keeping actual compute cost per request closer to that of a much smaller dense model, since only a fraction of parameters activate per input.