What Is AdamW? The Refinement Behind Most LLM Training Runs
A small but consequential fix to the Adam optimizer, correcting how it interacts with weight decay — now the more common choice for training large models.
A targeted fix to how Adam handles weight decay
AdamW is a modification of the Adam optimizer that changes how weight decay — a regularization technique — interacts with Adam's adaptive step sizing, decoupling the two so they don't interfere with each other the way they subtly did in the original Adam formulation.
Why this small fix mattered in practice
In standard Adam, weight decay gets tangled up with the same adaptive scaling used for the main gradient update, which weakens its intended regularizing effect in a way that isn't obvious just from reading the algorithm. AdamW separates these two mechanisms cleanly, restoring weight decay's intended, more predictable regularizing behavior.
Why it became the more common choice for large models
Empirically, AdamW has been found to produce better-generalizing models than plain Adam in many large-scale training setups, which is why it has become the more common optimizer specifically for training large language models, even though the difference from Adam is a relatively small implementation detail.
Frequently Asked Questions
What is AdamW?
A modification of the Adam optimizer that decouples weight decay from Adam's adaptive step sizing, fixing a subtle interaction that weakened weight decay's intended regularizing effect in the original Adam formulation.
Why is AdamW preferred over Adam for training large language models?
AdamW has empirically been found to produce better-generalizing models in many large-scale training setups, making it the more common optimizer choice for LLM training despite being a relatively small change to the original algorithm.