What Is a Tokenizer? How Text Becomes Numbers a Model Can Use
Before a model can process any text, it has to be broken into tokens and converted to numbers — the tokenizer is the component that does that conversion.
Breaking text into chunks a model understands
A tokenizer splits raw text into smaller chunks — tokens — that might be whole words, parts of words, or individual characters, then maps each one to a numeric ID from a fixed vocabulary. A model never actually sees text directly; every input first passes through this conversion.
Why not just split on spaces
Splitting purely on whitespace would create an enormous vocabulary (every word form treated as entirely separate) and would completely break on misspellings, made-up words, or languages without clear spacing. Modern tokenizers instead use algorithms like byte pair encoding to find a practical middle ground between individual characters and whole words.
The tokenizer is fixed after training
A model's tokenizer is established before training begins and generally can't be changed afterward without retraining — every token ID the model has ever learned about is tied to that specific vocabulary, which is why providers rarely swap tokenizers mid-generation of a model family.
Frequently Asked Questions
What does a tokenizer do?
It converts raw text into a sequence of tokens, then maps each token to a numeric ID from a fixed vocabulary — the first step in turning text into something a model can actually process.
Why can't a model just read raw text directly?
Neural networks operate on numbers, not characters or words directly. The tokenizer is what bridges that gap, converting text into the numeric token IDs the model was trained on.