What Is Gradient Checkpointing? Trading Compute for Memory During Training
A technique that saves memory during training by deliberately recomputing certain values instead of storing them — trading extra computation for a smaller memory footprint.
Recomputing instead of storing
Gradient checkpointing saves memory during training by deliberately not storing every intermediate value computed during the forward pass, and instead recomputing those values again later, when they're needed for backpropagation — trading extra computation for meaningfully reduced memory usage.
Why this specific tradeoff makes sense
Storing every intermediate value from a deep network's forward pass consumes a large amount of memory, and memory is frequently the binding constraint on how large a model or batch fits on available hardware. Recomputing some values instead of storing all of them can meaningfully lower memory needs at the cost of noticeably more computation.
Where it's used in practice
Gradient checkpointing is commonly combined with other memory-saving techniques like mixed precision training and gradient accumulation — all aimed at fitting a larger model or batch size onto available hardware than would otherwise be possible.
Frequently Asked Questions
What is gradient checkpointing?
A technique that saves memory during training by not storing every intermediate forward-pass value, instead recomputing them later when needed for backpropagation — trading extra computation for reduced memory usage.
Why would a team choose to recompute values instead of storing them?
Memory is frequently the binding constraint on how large a model or batch fits on available hardware. Recomputing some values instead of storing them all can meaningfully lower memory needs at the cost of extra computation.