What Is Batch Normalization? Stabilizing Training Using Batch Statistics
An earlier normalization technique that stabilizes training by rescaling values based on statistics computed across the current batch — with a key limitation for language models.
Rescaling based on the current batch
Batch normalization rescales the values passing through a layer based on statistics — mean and variance — computed across the current training batch, keeping those values within a consistent range to help stabilize training, similar in goal to layer normalization.
The key difference from layer normalization
Batch normalization computes its statistics across the examples in a batch, which means its behavior depends on batch size and composition, and it behaves inconsistently between training (using batch statistics) and inference (using a single example) — layer normalization avoids this by normalizing per-example instead.
Why language models mostly use layer normalization instead
Batch normalization's dependence on batch statistics is awkward for sequence models processing variable-length inputs, and for generation, which often happens one token or one example at a time. This is a major reason transformer-based language models standardized on layer normalization rather than batch normalization.
Frequently Asked Questions
What is batch normalization?
A technique that rescales values passing through a layer based on statistics computed across the current training batch, helping stabilize training in a way similar in goal to layer normalization.
Why do language models typically use layer normalization instead of batch normalization?
Batch normalization's dependence on batch statistics is awkward for variable-length sequences and for generation happening one token at a time, which is why transformer-based language models standardized on layer normalization instead.