What Is Layer Normalization? Keeping Training Stable Across Deep Networks
A behind-the-scenes technique that rescales values inside a network at each layer, keeping training stable enough for very deep models to actually converge.
Rescaling values as they flow through the network
Layer normalization rescales the values passing through a given layer so they stay within a consistent, predictable range — regardless of how large or small they happened to grow in earlier layers. It's applied per-example, independent of what else is in the current training batch.
Why unstable value ranges are a problem
Without normalization, the scale of values can drift wildly as they pass through many stacked layers, which makes gradient descent updates unpredictable and training prone to instability or outright failure. Keeping the ranges consistent gives training a much steadier signal to work with.
A standard fixture in transformers
Every transformer layer applies layer normalization, typically paired with a residual connection around the same sub-layer. Together, these two unglamorous mechanisms are a large part of why transformer training at massive depth and scale is even feasible.
Frequently Asked Questions
What is layer normalization?
A technique that rescales the values passing through a network layer to keep them within a consistent, predictable range, which helps keep training stable as networks get deeper.
What happens without layer normalization?
Value ranges can drift unpredictably as data passes through many stacked layers, making training updates unstable and increasing the risk that training fails to converge properly.