What Is Byte Pair Encoding (BPE)? How Modern Tokenizers Are Built
The algorithm behind most LLM tokenizers — it learns a vocabulary of common chunks directly from data, striking a balance between whole words and single characters.
Starting small, merging up
Byte pair encoding starts with individual characters (or bytes) as its base units, then repeatedly finds the most frequently occurring adjacent pair in its training text and merges them into a single new unit — repeating this process thousands of times to build up a vocabulary of common chunks, from single letters up to frequent whole words.
Why this beats a fixed word list
Because it's built from data rather than a predefined dictionary, BPE naturally handles rare words, typos, and made-up terms by falling back to smaller familiar chunks or individual characters, rather than treating anything outside a fixed word list as completely unrecognizable.
The result: a data-driven vocabulary
The final set of merged chunks becomes the model's vocabulary — common English words often end up as single tokens, while rarer or foreign words get split into a few smaller pieces. This is why token count doesn't map cleanly to word count, and why unusual text tends to use more tokens per word than common text.
Frequently Asked Questions
What is byte pair encoding?
An algorithm that builds a tokenizer's vocabulary by starting with individual characters and repeatedly merging the most frequent adjacent pairs, producing a data-driven set of common chunks rather than a fixed word list.
Why does rare or unusual text use more tokens?
Common words tend to end up as single tokens because they appeared frequently during vocabulary building, while rare or unfamiliar words get split into several smaller sub-word tokens instead.