What Is a Vocabulary in NLP? The Fixed Set of Tokens a Model Knows
Every model has a finite, fixed list of tokens it can ever produce — its vocabulary. Here's what determines that list and why its size matters.
A closed, fixed set
A model's vocabulary is the complete, fixed set of tokens — established by its tokenizer before training — that it can ever read or generate. Typical vocabularies contain tens of thousands to over a hundred thousand entries, covering common words, word pieces, punctuation, and special control tokens.
Why vocabulary size is a real design choice
A larger vocabulary lets more common words fit into a single token (making sequences shorter and cheaper to process), but it also means more parameters spent just representing the vocabulary itself. A smaller vocabulary saves parameters there, at the cost of needing more tokens — and more compute — per typical piece of text.
What happens outside the vocabulary
A well-built vocabulary (typically via byte pair encoding) is designed so nothing is ever truly "unrecognized" — unfamiliar text just gets broken into more, smaller known pieces rather than failing outright, all the way down to individual characters or bytes if necessary.
Frequently Asked Questions
What is a vocabulary in an AI language model?
The complete, fixed set of tokens a model can read or generate, established by its tokenizer before training — typically tens of thousands to over a hundred thousand entries.
Does vocabulary size affect model performance?
It affects efficiency more directly than raw capability — a larger vocabulary means shorter token sequences for typical text but more parameters spent representing the vocabulary itself, and vice versa for a smaller one.