What Is Causal Language Modeling? The Training Objective Behind Modern LLMs
The core task nearly every modern LLM is trained on: given everything so far, predict what comes next — nothing more, nothing less.
Predict what comes next, using only what came before
Causal language modeling trains a model on a single task: given a sequence of tokens so far, predict the next one — using only the tokens that came before, never anything that comes after. This "causal" constraint (past can influence future, never the reverse) matches exactly how text actually gets generated, one token at a time.
Why such a simple objective produces such capable models
Predicting the next token accurately across a large enough and varied enough body of text turns out to require implicitly learning grammar, facts, reasoning patterns, and style — the objective sounds narrow, but succeeding at it broadly requires a huge amount of the underlying knowledge a useful assistant needs.
The foundation everything else builds on
Causal language modeling is exactly what happens during pretraining — it's the base objective a model learns before any further stage like supervised fine-tuning or RLHF reshapes its behavior into something more directly useful as an assistant.
Frequently Asked Questions
What is causal language modeling?
A training objective where a model predicts the next token in a sequence using only the tokens that came before it, never anything after — matching exactly how text is generated one token at a time.
Is causal language modeling the same as pretraining?
Pretraining is the stage where causal language modeling actually happens — it's the specific training objective used during pretraining, before further stages like fine-tuning or RLHF reshape the model's behavior.