What Is Weight Decay? A Simple Technique to Discourage Overfitting
A small, constant nudge applied during every training step, gently pulling parameters toward smaller values — a simple but effective defense against overfitting.
A constant, gentle pull toward smaller values
Weight decay adds a small penalty during training that gently pulls every parameter toward smaller values with each update, in addition to the normal update driven by the loss function — a constant, mild counterforce against parameters growing unnecessarily large.
Why smaller parameter values tend to generalize better
Very large parameter values are often associated with a model that has memorized specific quirks of its training data rather than learning genuinely general patterns — the hallmark of overfitting. Discouraging large values pushes the model toward simpler, more broadly generalizable solutions.
A form of regularization
Weight decay is one specific technique within the broader category of regularization — methods that intentionally constrain a model during training to favor solutions that generalize well, rather than optimizing purely and exclusively for the lowest possible training error.
Frequently Asked Questions
What is weight decay?
A small penalty added during training that gently pulls every parameter toward smaller values with each update, discouraging parameters from growing unnecessarily large.
How does weight decay help prevent overfitting?
Very large parameter values are often associated with a model that has memorized specific training data quirks rather than learning general patterns. Discouraging large values pushes the model toward simpler, more generalizable solutions.