Training
What Is Model Distillation?
How providers train small, fast models that punch above their weight by learning from a larger "teacher" model.
Teacher and student
Distillation is a training technique where a smaller "student" model is trained to mimic the outputs (or internal behavior) of a larger, more capable "teacher" model — rather than, or in addition to, learning directly from raw human-labeled data. Instead of just learning "this is the right answer," the student can learn from the teacher's full probability distribution over possible answers, which carries more signal than a single correct label would.
Why providers bother
Training a frontier-scale model is enormously expensive (see Pretraining: How Models Learn Language), but running one for every single user request is expensive too, and often overkill — plenty of tasks (quick classification, simple rewrites, fast chat responses) don't need frontier-level reasoning to handle well. Distilling a large model's capabilities down into a much smaller one lets a provider offer a fast, cheap tier of their product (often the free or "mini/lite" tier of a model family) that retains a surprising amount of the larger model's quality for common tasks.
The trade-off
A distilled model is rarely a full replacement for its teacher — it typically trails on the hardest reasoning tasks, longer-context understanding, and edge cases the teacher handles more reliably. The goal isn't to fully replicate the teacher, but to capture enough of its behavior that the much cheaper, much faster student is the right choice for the large majority of everyday requests, reserving the expensive frontier model for tasks that actually need it.