What Is a Residual Connection? Why Deep Networks Can Train at All
A simple architectural trick — adding a layer's input back to its output — that made it possible to train networks with far more layers than before.
A shortcut around each layer
A residual connection adds a layer's original input directly back onto its output, so each layer only has to learn a small adjustment ("the difference") rather than the entire desired transformation from scratch. This shortcut path runs alongside the layer's normal processing, not instead of it.
Why this helps training so much
Without this shortcut, very deep networks tend to suffer from vanishing gradients — the training signal weakens as it passes back through many layers during backpropagation. Residual connections give that signal a more direct path through the network, which is a big part of why architectures with dozens or hundreds of layers became trainable at all.
A standard part of the transformer block
Every transformer layer includes residual connections around both its attention and feed-forward sub-layers, alongside layer normalization. It's an unglamorous detail that's nonetheless essential to why transformer-based models can be stacked as deep as modern LLMs are.
Frequently Asked Questions
What is a residual connection?
An architectural shortcut that adds a layer's input directly to its output, letting each layer learn a small adjustment rather than a full transformation from scratch — making very deep networks much easier to train.
Why do deep neural networks need residual connections?
Without them, the training signal tends to weaken as it passes backward through many layers (vanishing gradients), making deep networks hard to train. Residual connections give that signal a more direct path.