Foundations
What Are Tokens?
The chunks of text an LLM actually reads and writes — and why they don't map 1:1 to words.
Not words, not letters — tokens
Every model in this section reads text as tokens, not as whole words or individual characters. A tokenizer breaks input into chunks somewhere in between: common short words are often a single token, longer or rarer words get split into a few pieces, and punctuation and spacing usually get their own tokens too. "Tokenization" is often the first thing that happens to your prompt before the model ever sees it, and the last thing that happens to the model's own output before it reaches you as readable text.
Why not just use whole words?
If a model only understood complete words, it would need a separate slot in its vocabulary for every word in every language it supports — plus every made-up word, typo, product name, and piece of code it might ever encounter. Splitting words into smaller sub-word pieces lets a model represent a huge range of text with a fixed, manageable vocabulary (commonly tens of thousands of possible tokens), while still handling unfamiliar words by breaking them into familiar pieces.
Why tokens matter to you as a user
- Context windows are measured in tokens, not words. A "128K token context window" is not 128,000 words — depending on the language and content, 1,000 tokens is roughly 750 English words, so the usable word count is smaller than the token number suggests.
- Pricing is usually per-token. Most API pricing (and many consumer "usage limits") are based on tokens in and tokens out, which is why a request with a long input or output costs more. See What Is a Context Window? for more on the limit itself.
- Non-English text often uses more tokens per word. Because tokenizers are typically trained on datasets dominated by English, some other languages get split into more, smaller tokens for the same amount of meaning — which can make usage costs and effective context length vary by language.