What Is Compute-Optimal Training? Balancing Model Size and Data
For a fixed training budget, there's a specific balance between model size and data volume that gets the best result — and it's not always "bigger model."
Finding the best split for a fixed budget
Compute-optimal training is about finding, for a fixed amount of available training compute, the best balance between how large to make the model versus how much training data to use — since compute spent on one comes directly at the expense of the other, given a fixed total budget.
Why "just make it bigger" isn't automatically optimal
Research into this tradeoff found that many earlier large models were actually undertrained relative to their size — they would have performed better with a comparatively smaller model trained on proportionally more data, for the exact same total compute budget, rather than maximizing parameter count alone.
Why this reshaped how labs plan training runs
This finding shifted industry practice toward using scaling laws to calculate the compute-optimal model size and data volume jointly for a given budget, rather than picking an ambitious parameter count first and training on whatever data happens to be available.
Frequently Asked Questions
What is compute-optimal training?
Finding, for a fixed amount of training compute, the best balance between model size and training data volume — since spending compute on one comes directly at the expense of the other within a fixed total budget.
Does a bigger model always perform better for the same compute budget?
Not necessarily — research found many earlier large models were undertrained relative to their size, and would have performed better as a smaller model trained on proportionally more data for the same total compute.