What Is a Context Window?

The model's working memory

A context window is the maximum number of tokens a model can consider at once — your prompt, any files or chat history you've included, and the response it generates all have to fit inside this one budget. Ask a model something that requires information outside that window (an earlier part of a very long conversation, for instance) and it simply can't see it anymore, no matter how important it was.

Why context windows used to be so small

Early transformer-based models had context windows in the low thousands of tokens, because the self-attention mechanism at the heart of the architecture (see How Transformers Work) gets more computationally expensive as the input gets longer — historically, quadratically so. Doubling the context length could mean roughly quadrupling the compute needed to process it, which made very long context windows impractical for a long time.

What changed

Newer architectural tricks — more efficient attention variants, better memory management during inference, and techniques that let models retrieve relevant context instead of re-processing all of it — have pushed context windows from a few thousand tokens to hundreds of thousands, and in some cases over a million, without costs scaling the same brutal way they used to.

Why the number on the box isn't the whole story

A large context window doesn't guarantee a model uses everything inside it equally well. It's well documented that many models are noticeably better at recalling information near the beginning or end of a long context than information buried in the middle — sometimes called a "lost in the middle" effect. A model with a 1M token window isn't automatically better at a task than one with 128K if the relevant information sits in that harder-to-recall middle stretch.