A monotonic but flexible fit
Isotonic regression fits the best nondecreasing step function to data. Unlike a line, it assumes only that the output rises or stays flat as the input grows, letting the shape follow the data freely.
How it is solved
The fit minimizes squared error subject to the monotonic constraint. The classic solver is the pool adjacent violators algorithm, which sweeps through points and merges any pair that violates the order into a shared average until everything is monotone.
The main use, calibration
- A classifier may rank examples well but output miscalibrated probabilities.
- Isotonic regression maps raw scores to true frequencies while preserving their order.
- It is more flexible than Platt scaling, which fits only a sigmoid, but needs more data to avoid overfitting the steps.
Cautions
- With little data the steps can overfit, so a sigmoid calibration may generalize better.
- It only enforces monotonicity, so it cannot fix a model whose scores rank examples in the wrong order.
Key idea
Isotonic regression fits a free form monotonic step function via pool adjacent violators. It calibrates classifier probabilities more flexibly than a sigmoid but needs more data to avoid overfitting.