Foundations
The Attention Mechanism, Explained Simply
Attention is the mechanism that lets a model weigh which earlier words actually matter for the one it's generating next — the core idea behind the transformer architecture.
The problem it solves
When generating a word, a model needs to know which of the (possibly thousands of) preceding words are actually relevant. "It" three paragraphs later might refer to a noun mentioned once, early on. Attention gives the model a learned way to look back across everything it's seen and weigh each part by relevance to what it's producing right now.
How it works, roughly
For each word being generated, attention computes a relevance score against every other word in the input, then blends their representations weighted by those scores — so highly relevant context contributes more, and irrelevant context contributes less. This happens fresh for every word generated, not just once for the whole input.
Why it was a breakthrough
Earlier architectures processed text strictly in order, one word affecting the next, which made it hard to connect information that was far apart in a sequence. Attention lets a model connect any two positions directly, however far apart, which is a big part of why the transformer architecture (see How Transformers Work) replaced those earlier approaches.
Frequently Asked Questions
What does "attention" mean in AI models?
A mechanism that lets a model weigh how relevant each earlier piece of text is to the word it's currently generating, rather than treating all prior context equally.
Is attention the same thing as a transformer?
No — attention is the key mechanism inside a transformer, but a transformer is the full architecture built around it (attention layers plus other components).