What Is Masked Language Modeling? An Alternative to Next-Token Prediction
Before decoder-only models dominated, many language models learned by filling in blanks scattered across a sentence — a genuinely different training objective.
Filling in the blanks
Masked language modeling trains a model by hiding random tokens throughout a piece of text and asking it to predict exactly what was hidden, using both the words before and after each blank. It's a fundamentally different objective from causal language modeling, which only ever looks backward.
Why looking both directions is useful
Because a masked token's context includes words on both sides, a model trained this way builds a representation of each word informed by its entire surrounding sentence, not just what came before it — which historically made this style of training strong for tasks like classification and understanding text, rather than generating it.
Why generation moved toward causal modeling instead
A model trained to fill in gaps using future context isn't naturally suited to generating text left-to-right, since during real generation there is no "future" text yet to look at. That mismatch is a big part of why chat-style generative models settled on causal, decoder-only training instead.
Frequently Asked Questions
What is masked language modeling?
A training approach where random tokens in a sentence are hidden and the model must predict them using both the surrounding words before and after each blank — a different objective than predicting only the next word.
Why don't most chat AI models use masked language modeling?
A model trained to fill in gaps using future context isn't naturally suited to left-to-right generation, since real generation has no future text available — which is why causal, decoder-only training became the standard for generative chat models.