[Paper Review] Fairness in Ranking: A Survey

Meike Zehlike, Ke Yang+1 more|arXiv (Cornell University)|Mar 25, 2021

Auction Theory and Applications66 references29 citations

TL;DR

A comprehensive survey of fairness in ranking across score-based, supervised learning-to-rank, and recommender/matching systems, with a unifying narrative of value frameworks and mitigation approaches.

ABSTRACT

In the past few years, there has been much work on incorporating fairness requirements into algorithmic rankers, with contributions coming from the data management, algorithms, information retrieval, and recommender systems communities. In this survey we give a systematic overview of this work, offering a broad perspective that connects formalizations and algorithmic approaches across subfields. An important contribution of our work is in developing a common narrative around the value frameworks that motivate specific fairness-enhancing interventions in ranking. This allows us to unify the presentation of mitigation objectives and of algorithmic techniques to help meet those objectives or identify trade-offs. In this survey, we describe four classification frameworks for fairness-enhancing interventions, along which we relate the technical methods surveyed in this paper, discuss evaluation datasets, and present technical work on fairness in score-based ranking. Then, we present methods that incorporate fairness in supervised learning, and also give representative examples of recent work on fairness in recommendation and matchmaking systems. We also discuss evaluation frameworks for fair score-based ranking and fair learning-to-rank, and draw a set of recommendations for the evaluation of fair ranking methods.

Motivation & Objective

• Provide a systematic overview of fairness in ranking across data management, algorithms, information retrieval, and recommender systems.
• Unify mitigation objectives and algorithmic techniques through a common narrative of fairness in ranking.
• Present four classification frameworks for fairness-enhancing interventions and relate them to surveyed methods.
• Discuss evaluation datasets and representative work on fairness in score-based ranking, LtR, and recommender systems.
• Offer recommendations for evaluating fair ranking methods and future research directions.

Proposed method

• Describe score-based ranking and formalize utility with top-k and group-specific utilities.
• Introduce fairness measures including proportional representation in top-k and prefixes, and discuss diversity/cov erage concepts.
• Present evaluation approaches for fairness via probability-based and exposure-based definitions.
• Explain supervised learning-to-rank (LtR) and its training/testing pipeline, including NDCG and MAP as accuracy metrics.
• Discuss fairness interventions in LtR to mitigate bias learned from training data.
• Provide representative examples of fairness in recommender systems and matching.

Experimental results

Research questions

• RQ1 What are the normative goals and practical mitigation objectives used to achieve fairness in ranking across subfields?
• RQ2 How can fairness interventions be categorized and connected across score-based ranking, LtR, and recommender systems?
• RQ3 What evaluation frameworks and datasets support fair ranking, and what recommendations emerge for evaluating fair ranking methods?
• RQ4 How do population-level and exposure-based fairness notions compare in the ranking context?

Key findings

• The survey provides a broad, unified perspective linking fairness concepts across subfields of ranking.
• It outlines four classification frameworks for fairness-enhancing interventions and connects them to technical methods.
• It discusses both score-based and LtR fairness, and highlights representative fairness work in recommender systems and matching.
• It emphasizes evaluation frameworks, datasets, and practical recommendations for evaluating fair ranking methods.
• It clarifies that fairness in ranking is value-laden and context-dependent, not purely a technical construct.