Squared error for probabilities
The Brier score measures the quality of probabilistic predictions for binary outcomes. It is simply the mean squared difference between the predicted probability and the actual outcome, where the outcome is one for positive and zero for negative.
How to read it
- A perfect prediction scores zero.
- The worst possible confident wrong prediction scores one.
- Lower is always better.
Because it squares the gap, the Brier score punishes confident wrong predictions but more gently than log loss, which explodes toward infinity.
What it captures
The Brier score blends two qualities.
- Calibration, whether the stated probabilities match real frequencies.
- Refinement, whether the model pushes probabilities toward zero or one when it can.
A model that hedges everything at fifty percent scores poorly, and so does a confidently miscalibrated one.
Key idea
The Brier score is the mean squared error between predicted probabilities and outcomes. It rewards both calibration and decisiveness, and unlike log loss it bounds the penalty for confident mistakes.