[Paper Review] Assessing top-$k$ preferences

Charles L. A. Clarke, Alexandra Vtyurina+1 more|arXiv (Cornell University)|Jul 22, 2020

Mobile Crowdsensing and Crowdsourcing45 references4 citations

TL;DR

This paper proposes a new evaluation method for ranking systems using partial top-$k$ preference judgments, where assessors compare only the top items rather than full rankings. By measuring compatibility between a system's output and an ideal ranking derived from these preferences, the method detects subtle improvements in modern neural rankers that NDCG misses, offering a more sensitive and practical alternative to traditional graded evaluation.

ABSTRACT

Assessors make preference judgments faster and more consistently than graded judgments. Preference judgments can also recognize distinctions between items that appear equivalent under graded judgments. Unfortunately, preference judgments can require more than linear effort to fully order a pool of items, and evaluation measures for preference judgments are not as well established as those for graded judgments, such as NDCG. In this paper, we explore the assessment process for partial preference judgments, with the aim of identifying and ordering the top items in the pool, rather than fully ordering the entire pool. To measure the performance of a ranker, we compare its output to this preferred ordering by applying a rank similarity measure.We demonstrate the practical feasibility of this approach by crowdsourcing partial preferences for the TREC 2019 Conversational Assistance Track, replacing NDCG with a new measure named "compatibility". This new measure has its most striking impact when comparing modern neural rankers, where it is able to recognize significant improvements in quality that would otherwise be missed by NDCG.

Motivation & Objective

• To address the lack of well-established evaluation measures for preference judgments in information retrieval.
• To improve sensitivity in detecting performance differences among modern neural rankers, especially when NDCG fails to capture meaningful gains.
• To reduce assessment cost and effort by focusing only on top-$k$ items rather than full rankings.
• To develop a practical, scalable method for offline evaluation using crowdsourced partial preferences.
• To demonstrate that preference-based evaluation can better reflect real user preferences than graded judgments.

Proposed method

• Focus on partial preferences: assessors judge only the top-$k$ items per query, forming a weak ordering of the best results.
• Define an ideal ranking set as all permutations of the top-$k$ items that are consistent with the partial preferences, excluding non-relevant items.
• Use Rank Biased Overlap (RBO) to compute similarity between the system’s actual ranking and each ideal ranking in the set.
• Compute compatibility as the maximum RBO similarity between the system’s output and any ideal ranking in the set.
• Replace traditional NDCG with compatibility as the primary evaluation measure in offline IR evaluation.
• Apply the method to the TREC 2019 Conversational Assistance Track, using crowdsourced top-5 preferences and original graded judgments as baselines.

Experimental results

Research questions

• RQ1 Can partial top-$k$ preference judgments provide a more sensitive evaluation metric than NDCG for modern neural rankers?
• RQ2 How does compatibility compare to NDCG in detecting meaningful performance differences among rankers?
• RQ3 Can compatibility effectively identify improvements in rankers that NDCG fails to capture?
• RQ4 What is the practical feasibility of using crowdsourced partial preferences for large-scale IR evaluation?
• RQ5 Does focusing on top-$k$ preferences improve alignment with actual user click behavior and preference patterns?

Key findings

• Compatibility detected significant performance differences among the top four runs in the TREC 2019 CAsT track that NDCG@3 failed to distinguish.
• When using only crowdsourced top-5 answers as ideal rankings, compatibility achieved 73.3% sensitivity in detecting ranker differences, higher than with graded judgments alone.
• With only the single best answer used as the ideal, compatibility still maintained a relative ordering of top runs, though sensitivity dropped to 55.2%.
• Neural rankers showed measurable improvements under compatibility that were undetected by NDCG, indicating compatibility’s superior sensitivity to subtle quality gains.
• The method is practically feasible and scalable, as demonstrated through crowdsourcing on a real-world IR evaluation task.
• Simplifying initial graded assessment to three categories (relevant, related, not relevant) could streamline the process while preserving evaluation quality.