What Is Label Smoothing? Softening Overly Confident Training Targets
A small tweak to how training targets are represented, discouraging a model from becoming overconfident and often improving how well it actually generalizes.
Softening an all-or-nothing target
Label smoothing slightly softens a training target — instead of the correct answer being represented as absolute certainty (100%) and every other option as impossible (0%), a small amount of probability gets deliberately redistributed away from the "correct" answer toward the other options, making the target less extreme.
Why softening the target actually helps
Training a model to output absolute, 100% certainty for the correct answer tends to push it toward being overconfident in its predictions generally, including on cases where it's actually wrong. Label smoothing discourages that extreme overconfidence, often improving how well the resulting model calibrates and generalizes to new data.
Where this technique commonly applies
Label smoothing is widely used in classification tasks and has also seen use in language model training, as a lightweight, simple technique to improve calibration — how well a model's stated confidence actually reflects its true accuracy — without requiring any change to the model's architecture itself.
Frequently Asked Questions
What is label smoothing?
A technique that slightly softens a training target, redistributing a small amount of probability away from the "correct" answer toward other options, rather than representing the target as absolute 100% certainty.
Why does label smoothing help?
Training toward absolute certainty tends to push a model toward general overconfidence, including on cases where it's wrong. Softening the target discourages that overconfidence, often improving how well the model calibrates and generalizes.