The trap in click logs
Items shown at the top of a list get more clicks simply because users look there first. This is position bias: an item at slot one and the same item at slot ten receive very different click rates regardless of true relevance. Training naively on clicks teaches the model that high positions are good, a self reinforcing loop.
Why it matters
- The model learns to favor whatever was already ranked high, not what is truly best.
- New or buried good items never get a fair chance to prove themselves.
- Offline metrics look fine while the system slowly ossifies.
Modeling the bias
A common fix is the examination hypothesis: the chance of a click equals the chance the user examined the position times the chance the item is relevant. If examination depends only on position, you can separate the two.
Correction techniques
- Inverse propensity weighting reweights each logged click by one over its position examination probability, removing the slot advantage.
- Position as a feature feeds the slot into training, then sets it to a neutral value at serving so the model uses relevance, not position.
- Randomization occasionally shuffles results to estimate examination curves directly.
The payoff
Correcting position bias lets the ranker learn true relevance, surfacing items that were unfairly buried and breaking the feedback loop.
Key idea
Position bias makes top slots look better than they are; modeling examination and using inverse propensity weighting separates true relevance from the slot advantage.