What Is Mixed Precision Training? Speeding Up Training With Lower Precision Numbers
Using lower-precision numbers for parts of training can dramatically speed things up — mixed precision training does this selectively, without sacrificing overall accuracy.
Using lower precision where it's safe to
Mixed precision training uses lower-precision numbers (fewer bits per value) for most of the computation during training, while selectively keeping higher precision for the specific operations where reduced precision would meaningfully hurt accuracy — a deliberate, targeted mix rather than uniformly lowering precision everywhere.
Why lower precision speeds things up so much
Lower-precision arithmetic runs significantly faster on modern GPUs, which are specifically built with hardware optimized for it, and it uses less memory per value — letting more computation happen per second and larger batches fit in the same amount of hardware memory.
Why it doesn't noticeably hurt final model quality
Neural network training turns out to be fairly tolerant of reduced numeric precision in most places, similar to the tolerance that makes quantization viable after training — as long as a few precision-sensitive operations are kept at higher precision, overall training quality is largely preserved.
Frequently Asked Questions
What is mixed precision training?
A training approach that uses lower-precision numbers for most computation, while selectively keeping higher precision for specific operations where reduced precision would meaningfully hurt accuracy.
Why does mixed precision training speed things up?
Lower-precision arithmetic runs significantly faster on modern GPU hardware built to optimize for it, and uses less memory per value, allowing more computation per second and larger batches within the same hardware memory.