What Is Gradient Explosion? When Training Updates Spiral Out of Control
A training failure mode where gradients grow uncontrollably large as they pass backward through a network, destabilizing training entirely.
Gradients growing uncontrollably large
Gradient explosion happens when gradients grow to extremely large values as they propagate backward through a deep network during backpropagation — each layer's computation compounding on the last until the resulting values become impractically, destructively large.
Why this destabilizes training so severely
An exploded gradient produces a wildly oversized parameter update, which can push a model's parameters to nonsensical values in a single step — often visible directly as a severe loss spike or, in extreme cases, numerical values that become undefined entirely (NaN), effectively breaking training outright.
How it's typically prevented
Gradient clipping is the most direct, common defense, capping gradient magnitude before it can cause damage. Careful weight initialization and layer normalization also help by keeping the values flowing through the network in a more stable, controlled range from the start.
Frequently Asked Questions
What is gradient explosion?
A training failure mode where gradients grow to extremely large values as they propagate backward through a deep network, producing wildly oversized parameter updates that can destabilize or break training entirely.
How is gradient explosion typically prevented?
Gradient clipping is the most direct defense, capping gradient magnitude before it causes damage. Careful weight initialization and layer normalization also help by keeping values flowing through the network more stable from the start.