[Paper Review] Overview of the TREC 2019 Fair Ranking Track

Asia J. Biega, Fernando Díaz+2 more|arXiv (Cornell University)|Mar 25, 2020

Information Retrieval and Search Behavior10 references4 citations

TL;DR

This paper introduces the TREC 2019 Fair Ranking Track, a benchmark for evaluating retrieval systems on fairness to content providers alongside relevance. It proposes a reranking task using academic abstracts from Semantic Scholar, with fairness measured via exposure metrics under unknown group definitions (e.g., institutional prestige, h-index), and reports that systems like fair_LambdaMART and uognleMaxUtil achieved high fairness and utility trade-offs across diverse groupings.

ABSTRACT

The goal of the TREC Fair Ranking track was to develop a benchmark for evaluating retrieval systems in terms of fairness to different content providers in addition to classic notions of relevance. As part of the benchmark, we defined standardized fairness metrics with evaluation protocols and released a dataset for the fair ranking problem. The 2019 task focused on reranking academic paper abstracts given a query. The objective was to fairly represent relevant authors from several groups that were unknown at the system submission time. Thus, the track emphasized the development of systems which have robust performance across a variety of group definitions. Participants were provided with querylog data (queries, documents, and relevance) from Semantic Scholar. This paper presents an overview of the track, including the task definition, descriptions of the data and the annotation process, as well as a comparison of the performance of submitted systems.

Motivation & Objective

• Address the imbalance in content discoverability for underrepresented authors in information retrieval systems.
• Develop a standardized evaluation framework for fairness in ranking that complements traditional relevance metrics.
• Create a benchmark dataset with annotated author group memberships and relevance labels for fair ranking evaluation.
• Encourage development of reranking systems robust to multiple, unknown group definitions at test time.
• Promote fairness metrics that measure long-term exposure of authors across ranking sequences, not just individual rankings.

Proposed method

• Define a browsing model-based exposure metric that computes expected exposure of authors based on ranking position and user stopping probability.
• Use click logs from Semantic Scholar to derive binary relevance labels, assuming clicks indicate relevance.
• Apply a reranking task where systems re-rank document sets per query to optimize for both relevance and fairness.
• Measure fairness using unfairness metrics derived from exposure disparities across predefined author groups.
• Implement macro- and micro-amortized evaluation across queries to assess system performance at scale.
• Use a protocol where group definitions are hidden until after submission to test system robustness to varying groupings.

Experimental results

Research questions

• RQ1 How can fairness in ranking be measured in a way that accounts for long-term exposure of content producers?
• RQ2 To what extent can reranking systems correct for biases in initial retrieval results that favor dominant authors or institutions?
• RQ3 How do different fairness-aware algorithms perform across multiple, unknown group definitions at test time?
• RQ4 What is the trade-off between relevance and fairness in academic search ranking under varying grouping schemes?
• RQ5 How do biases in click-based relevance labeling affect the fairness evaluation of ranking systems?

Key findings

• The fair_LambdaMART system achieved the highest expected utility (0.6599) and moderate unfairness (0.0741) under the IMF-level group definition with two groups.
• The uognleMaxUtil system achieved the highest expected utility (0.6741) and moderate unfairness (0.0799) under the same group definition.
• Systems like QUARTZ-e0.00100 and QUARTZ-e0.00200 showed strong fairness performance with unfairness values around 0.035 under the IMF-level grouping.
• Under the h-index grouping with four groups, fair_LambdaMART achieved an unfairness of 0.0855 and utility of 0.6599, indicating a notable trade-off.
• The uognleMaxUtil system achieved the highest utility (0.6741) and moderate unfairness (0.0656) under the h-index grouping, suggesting strong performance across group definitions.
• The results indicate that fairness-aware reranking can improve exposure equity without sacrificing relevance, especially when group definitions are not known in advance.