How Transformers Work

The architecture behind almost every LLM

Nearly every model you'll encounter in this section — GPT-4o, Claude, Gemini, Llama — is built on the same core architecture: the transformer, introduced in the 2017 paper "Attention Is All You Need." Before transformers, language models processed text one word at a time, in order, which made them slow to train and bad at connecting ideas that were far apart in a sentence.

Tokens, not words

A transformer doesn't read raw text. It first breaks input into tokens — chunks that are often smaller than a whole word (common words might be one token, rarer words split into several). Each token is converted into a vector of numbers the model can do math on. A model's "context window" is simply the maximum number of tokens it can hold in memory at once.

Self-attention

The key idea is self-attention: for every token, the model calculates how much every other token in the input matters to it. This lets the model connect "it" to the noun it refers to several sentences earlier, or weigh a word differently depending on surrounding context — all in parallel, rather than one token at a time. That parallelism is also why transformers train so much faster than the architectures that came before them.

Stacking layers

A single attention step isn't very powerful on its own. Real models stack dozens of transformer layers on top of each other, each refining the representation a little further. By the final layer, the model has built up a rich internal representation of the input that it can use to predict the next token — and the next, and the next, which is how it generates entire responses.

Why this matters for the models in this section

Every entry in the Models category is, underneath, a transformer trained on a massive amount of text (and increasingly images/audio too). The differences between them — size, training data, fine-tuning approach — are what the rest of this section covers.