When order matters
Search, recommendation, and question answering return ranked lists. Classification metrics do not capture whether the right item is near the top. Ranking metrics do.
Mean reciprocal rank
MRR focuses on the position of the first relevant result for each query.
- For one query, take the reciprocal of the rank of the first correct hit
- First place gives 1, second gives one half, third gives one third
- Average these reciprocals across all queries
So MRR rewards putting a correct answer as high as possible and ignores everything after the first hit.
When to use it
MRR fits tasks with essentially one right answer, like a knowledge base lookup or a navigational search where the user wants a single page.
- It is simple and interpretable
- It is blind to additional relevant items lower down
If multiple results can be relevant and their order matters throughout the list, MRR is too narrow and you should reach for NDCG or MAP instead.
Key idea
MRR scores only the position of the first correct result, averaged over queries. It shines for single answer tasks and ignores the rest of the list.