What Is the Vanishing Gradient Problem? When the Training Signal Fades Away
The opposite failure mode from exploding gradients: the training signal shrinks toward nothing as it passes backward through a deep network, stalling learning.
A training signal that shrinks toward nothing
The vanishing gradient problem happens when gradients shrink to near-zero values as they propagate backward through a deep network during backpropagation — the opposite failure mode from gradient explosion, where instead of growing uncontrollably, the training signal fades away to almost nothing.
Why a vanished gradient effectively stalls learning
A near-zero gradient produces a near-zero parameter update, meaning that specific parameter essentially stops learning at all, even though training is technically still running — this problem historically worsened as networks got deeper, since the shrinking effect compounds across each additional layer the signal has to pass through.
How modern architectures largely avoid it
Residual connections give the training signal a more direct shortcut path through the network, sidestepping much of the compounding shrinkage. Combined with careful weight initialization and normalization, this is a major reason modern deep architectures can train reliably at depths that would have suffered badly from vanishing gradients otherwise.
Frequently Asked Questions
What is the vanishing gradient problem?
A training failure mode where gradients shrink to near-zero values as they propagate backward through a deep network, effectively stalling learning for the affected parameters even though training is technically still running.
How do modern deep networks avoid vanishing gradients?
Residual connections give the training signal a more direct shortcut path through the network, sidestepping much of the compounding shrinkage, alongside careful weight initialization and normalization.