What Is Momentum in Optimization? Smoothing Out Noisy Training Updates
A technique that helps training keep moving consistently in a good direction, rather than reacting jerkily to every noisy, individual update.
Building up speed in a consistent direction
Momentum accumulates a running average of recent parameter updates, and uses that accumulated direction — not just the current step's raw gradient — to determine how each parameter actually moves, similar to how a rolling ball builds up speed and doesn't instantly change direction with every small bump.
Why this helps training converge faster and more smoothly
Individual training updates from small batches (as in stochastic gradient descent) are noisy and can point in slightly inconsistent directions from step to step. Momentum smooths that noise out, letting training move more consistently toward a good solution instead of jittering unpredictably.
Where momentum shows up today
Momentum isn't usually used entirely on its own anymore — it's built directly into modern optimizers like Adam, which combines momentum-style smoothing with per-parameter adaptive step sizing, rather than requiring it to be added as a completely separate technique.
Frequently Asked Questions
What is momentum in optimization?
A technique that accumulates a running average of recent parameter updates and uses that smoothed direction, rather than just the current step's raw gradient, to determine how parameters move — reducing noisy, jittery training updates.
Is momentum a separate technique from modern optimizers?
Not usually applied on its own anymore — it's built directly into modern optimizers like Adam, which combine momentum-style smoothing with adaptive, per-parameter step sizing in a single algorithm.