What Is Gradient Clipping? A Safety Cap on Training Updates
A simple safeguard that caps how large any single training update is allowed to be, preventing one bad batch from derailing an entire training run.
Capping how large an update can be
Gradient clipping sets a maximum limit on the size of the gradient used in a single training update — if the calculated gradient exceeds that limit, it gets scaled down before being applied, rather than being used at its full, uncapped size.
Why occasional huge gradients are a real risk
Certain unusual training batches can produce an unusually large gradient, which would otherwise cause an oversized, destabilizing parameter update — sometimes triggering a loss spike or even derailing an entire training run. Gradient clipping acts as a safety cap against exactly that failure mode.
A cheap safeguard for long, expensive training runs
Because a frontier training run can take weeks or months and cost an enormous amount, a cheap safeguard against a single bad batch derailing the entire process is a worthwhile trade — gradient clipping is close to standard practice in large-scale training specifically for this reason.
Frequently Asked Questions
What is gradient clipping?
A technique that caps the maximum size of the gradient used in a training update, scaling it down if it exceeds a set limit, to prevent unusually large updates from destabilizing training.
Why does gradient clipping matter for large training runs?
A frontier training run can take weeks or months and cost enormous amounts of compute. Gradient clipping is a cheap safeguard against a single unusual batch derailing that entire expensive process.