The Bias Variance Tradeoff Revisited
Expected prediction error on unseen data splits into three parts: bias, variance, and irreducible noise. Understanding this split tells you whether a model is too simple or too complex.
The three components
- Bias is the error from wrong assumptions. A model that is too simple, like a straight line for curved data, has high bias and underfits.
- Variance is the error from sensitivity to the training sample. A model that is too flexible memorizes noise and changes wildly across datasets.
- Irreducible noise comes from randomness in the data and cannot be removed by any model.
The tradeoff
As you increase model complexity, bias falls but variance rises. Total error usually forms a U shape with a sweet spot in the middle.
- Underfit region has high bias and low variance.
- Overfit region has low bias and high variance.
- The best model balances the two to minimize expected error.
Techniques like regularization, more data, and ensembling let you shift this balance deliberately rather than by accident.
Key idea
Expected error decomposes into bias, variance, and irreducible noise, and the best model balances bias against variance to sit at the bottom of the error U.