[Paper Review] Offline Comparison of Ranking Functions using Randomized Data

Aman Agarwal, Xuanhui Wang+3 more|arXiv (Cornell University)|Oct 11, 2018

Advanced Bandit Algorithms ResearchDecision Sciences13 references3 citations

TL;DR

This paper proposes Trunc-match and Rand-interleaving, two methods to improve data efficiency and comparison sensitivity in offline evaluation of ranking functions using uniformly randomized historical interaction data. By truncating ranked lists and leveraging interleaving to compare two models simultaneously, the methods significantly outperform traditional Direct-match in query retention and statistical power, with Rand-interleaving showing the highest sensitivity in detecting performance differences.

ABSTRACT

Ranking functions return ranked lists of items, and users often interact with these items. How to evaluate ranking functions using historical interaction logs, also known as off-policy evaluation, is an important but challenging problem. The commonly used Inverse Propensity Scores (IPS) approaches work better for the single item case, but suffer from extremely low data efficiency for the ranked list case. In this paper, we study how to improve the data efficiency of IPS approaches in the offline comparison setting. We propose two approaches Trunc-match and Rand-interleaving for offline comparison using uniformly randomized data. We show that these methods can improve the data efficiency and also the comparison sensitivity based on one of the largest email search engines.

Motivation & Objective

• To address the low data efficiency of Inverse Propensity Scores (IPS) in offline evaluation of ranked lists, especially when matching full lists is infeasible due to combinatorial explosion.
• To improve data efficiency and comparison sensitivity in offline evaluation by exploiting the fact that only pairwise comparison (which model is better) is needed, not pointwise metric estimation.
• To extend the use of interleaving—previously used only in online A/B testing—to offline evaluation using uniformly randomized data.
• To demonstrate that randomized data can be leveraged more effectively for offline comparison through tailored matching and interleaving strategies.

Proposed method

• Trunc-match improves matching efficiency by truncating recorded ranked lists to their top-k items and comparing only those, increasing the chance of matching a new ranker’s top-k output.
• The method assumes that truncating uniformly randomized lists preserves uniform randomness, enabling unbiased evaluation on the truncated subset.
• Rand-interleaving constructs interleaved lists by taking the minimum rank from two competing rankers at each position, enabling direct comparison of their relative performance on the same query.
• The approach uses the interleaved list to estimate click-through rates, treating the interleaving as a form of paired comparison that increases sensitivity.
• Both methods are applied to a large-scale email search engine dataset with 1.03M queries, using resampling to compute confidence intervals and assess statistical significance.
• The evaluation compares Trunc-match and Rand-interleaving against the baseline Direct-match, measuring retention rate, mean reciprocal rank (MRR@k), and click counts with error bars.

Experimental results

Research questions

• RQ1 Can truncating randomized ranked lists improve data efficiency in offline evaluation of ranking functions?
• RQ2 Does interleaving two ranking functions in a single randomized list enhance comparison sensitivity compared to single-model matching?
• RQ3 How do Trunc-match and Rand-interleaving compare to Direct-match in terms of query retention and statistical power for detecting performance differences?
• RQ4 Can randomized data be used effectively for offline comparison without requiring full list matching or pointwise metric estimation?

Key findings

• Trunc-match retains approximately 1/k! of the queries for a given k, which matches theoretical expectations and significantly improves over Direct-match’s low retention rate.
• Rand-interleaving retains slightly more queries than Trunc-match due to the higher probability of matching at least one of the two models’ top-k items in each position.
• Direct-match shows overlapping error bars even at k=1, indicating poor data efficiency and low statistical power for comparison.
• Trunc-match improves upon Direct-match in both retention and comparison sensitivity, but Rand-interleaving achieves the best separation between the two rankers in terms of mean and standard error.
• The results confirm that interleaving enhances sensitivity not just by increasing data usage, but by fundamentally improving the ability to detect performance differences.
• The proposed methods are effective in a real-world email search setting with 1.03M queries, demonstrating practical utility for large-scale offline evaluation.