What Is a Pretraining Corpus? The Raw Text a Model Learns Language From
The massive body of text used for the initial pretraining stage — where it comes from, how large it typically is, and why its composition matters so much.
The raw material of pretraining
A pretraining corpus is the large body of text used during a model's initial pretraining stage — typically drawing from sources like web pages, books, code repositories, and academic papers, combined according to a defined data mixture before training begins.
Just how large "large" actually is
Frontier model pretraining corpora commonly reach into the trillions of tokens — an amount of text vastly beyond what any individual human could read in a lifetime — reflecting how directly model capability tends to scale with both data volume and quality, per observed scaling laws.
Why composition matters as much as size
Raw size alone doesn't guarantee a good corpus — deduplication, filtering for quality, and a deliberate mixture across sources all shape a corpus's actual usefulness far more than sheer token count alone.
Frequently Asked Questions
What is a pretraining corpus?
The large body of text used during a model's initial pretraining stage, typically combining web pages, books, code, and academic papers according to a deliberately defined mixture across sources.
How large is a typical pretraining corpus?
Frontier models commonly train on corpora reaching into the trillions of tokens — an amount of text far beyond what any individual could read in a lifetime, reflecting how model capability tends to scale with data volume and quality.