The tradeoff
Every model error decomposes into bias (wrong assumptions — underfitting) and variance (sensitivity to the training sample — overfitting).
How you spot it
- Underfitting: high training error and high validation error.
- Overfitting: low training error but high validation error — the gap is the tell.
Levers
Increase capacity or features to cut bias. Add regularization, dropout, or more data to cut variance. The art is finding the capacity where validation error bottoms out.
Key idea
You never optimize training error — you optimize generalization. A model that memorizes the training set has learned nothing useful.