What Is a Recurrent Neural Network (RNN)?
Before transformers, this was how AI processed sequences — reading one step at a time while carrying a memory forward. Here's why it was eventually replaced.
Processing one step at a time
A recurrent neural network (RNN) processes a sequence step by step, maintaining a running internal "state" that gets updated at each step and carried forward to the next. That running state is meant to summarize everything seen so far, letting the network use earlier context when processing later input.
The problem: long-range memory fades
In practice, information from many steps earlier tends to get diluted or lost as the running state keeps getting overwritten step after step — a problem sometimes described via vanishing gradients during training. That made RNNs weaker at connecting distant parts of a long sequence, exactly the kind of long-range dependency modern language tasks need.
Why transformers replaced them
Attention solved this by letting a model connect any two positions directly, regardless of distance, instead of relying on information surviving a long chain of sequential updates. That's the core reason the transformer architecture displaced RNNs as the default for language models.
Frequently Asked Questions
What is a recurrent neural network?
A neural network architecture that processes sequences step by step, carrying a running internal state forward from one step to the next to represent everything seen so far.
Why did transformers replace RNNs for language models?
RNNs struggle to retain information over long distances because their running state gets diluted with each step. Attention in transformers lets a model connect any two positions directly, which handles long-range context far better.