Training
Pretraining: How Models Learn Language
The first, most expensive training stage — where a model learns to predict text from a huge unlabeled corpus.
Learning by prediction, at massive scale
Pretraining is the first and by far the most computationally expensive stage of building an LLM. The task is deceptively simple: given a chunk of text, predict the next token. Repeated across trillions of tokens scraped from the public web, books, code repositories, and other sources, this single simple objective forces the model to implicitly learn grammar, facts, reasoning patterns, and coding conventions — none of which are directly labeled anywhere in the training data. Nobody tells the model "this is a fact" or "this is valid Python" — it has to infer all of that just from what makes accurate next-token predictions possible.
Why it's so expensive
Pretraining a frontier model involves running that next-token prediction task across a training set of trillions of tokens, adjusting billions of parameters a tiny amount after each batch. This is why frontier pretraining runs are reported to cost providers tens to hundreds of millions of dollars in compute, and why only a small number of organizations attempt it from scratch — most companies building "AI products" are fine-tuning or prompting an existing pretrained model rather than pretraining their own (see Fine-Tuning vs Prompting).
What you get at the end of pretraining
A freshly pretrained model — often called a "base model" — is genuinely impressive at completing text plausibly, but it is not yet a helpful assistant. It will continue a question with more questions in the same style rather than answering it, because "continue this text plausibly" is all it was ever trained to do. Turning a base model into something like ChatGPT or Claude requires the additional training stages covered in What Is RLHF?.