What Is the Bias-Variance Tradeoff? A Core Tension in Machine Learning
A model that's too simple systematically misses patterns; a model that's too complex fits noise instead of signal — a fundamental tension that runs through nearly all of machine learning.
Two different ways a model can go wrong
Bias is systematic error from a model being too simple to capture the real underlying pattern in the data. Variance is error from a model being so sensitive to its specific training data that it fits noise along with genuine signal — the same underlying model, seen from two different angles of failure.
Why you generally can't minimize both simultaneously
A model too simple to capture real patterns (high bias) tends to be very stable across different training sets (low variance). A model complex enough to fit training data very closely (low bias) tends to be very sensitive to the specific quirks of whatever data it happened to see (high variance) — reducing one typically increases the other.
Why this framing matters practically
Overfitting is essentially the high-variance failure mode; underfitting is the high-bias one. Regularization, cross-validation, and choosing appropriate model size and capacity are all, in different ways, attempts to find a good working balance between these two competing sources of error.
Frequently Asked Questions
What is the bias-variance tradeoff?
The fundamental tension where reducing bias (systematic error from a model too simple to capture real patterns) tends to increase variance (error from a model too sensitive to its specific training data's noise), and vice versa.
How does the bias-variance tradeoff relate to overfitting?
Overfitting is essentially the high-variance failure mode — a model too sensitive to training data specifics. Underfitting is the high-bias counterpart. Techniques like regularization aim to find a good balance between the two.