[Paper Review] MergeDTS: A Method for Effective Large-Scale Online Ranker Evaluation

Chang Li, Ilya Markov+2 more|arXiv (Cornell University)|Dec 11, 2018

Advanced Bandit Algorithms Research55 references4 citations

TL;DR

MergeDTS is a novel algorithm for large-scale online ranker evaluation that combines divide-and-conquer batching with Thompson Sampling to efficiently identify the optimal ranker under the Condorcet assumption. It significantly outperforms state-of-the-art dueling bandit methods like MergeRUCB in both regret and computational efficiency, especially in large-scale settings with hundreds or thousands of rankers.

ABSTRACT

Online ranker evaluation is one of the key challenges in information retrieval. While the preferences of rankers can be inferred by interleaving methods, the problem of how to effectively choose the ranker pair that generates the interleaved list without degrading the user experience too much is still challenging. On the one hand, if two rankers have not been compared enough, the inferred preference can be noisy and inaccurate. On the other, if two rankers are compared too many times, the interleaving process inevitably hurts the user experience too much. This dilemma is known as the exploration versus exploitation tradeoff. It is captured by the $K$-armed dueling bandit problem, which is a variant of the $K$-armed bandit problem, where the feedback comes in the form of pairwise preferences. Today's deployed search systems can evaluate a large number of rankers concurrently, and scaling effectively in the presence of numerous rankers is a critical aspect of $K$-armed dueling bandit problems. In this paper, we focus on solving the large-scale online ranker evaluation problem under the so-called Condorcet assumption, where there exists an optimal ranker that is preferred to all other rankers. We propose Merge Double Thompson Sampling (MergeDTS), which first utilizes a divide-and-conquer strategy that localizes the comparisons carried out by the algorithm to small batches of rankers, and then employs Thompson Sampling (TS) to reduce the comparisons between suboptimal rankers inside these small batches. The effectiveness (regret) and efficiency (time complexity) of MergeDTS are extensively evaluated using examples from the domain of online evaluation for web search. Our main finding is that for large-scale Condorcet ranker evaluation problems, MergeDTS outperforms the state-of-the-art dueling bandit algorithms.

Motivation & Objective

• To address the scalability challenge in online ranker evaluation when thousands of rankers are involved.
• To resolve the exploration-exploitation trade-off in large-scale dueling bandit problems under the Condorcet assumption.
• To develop a method that maintains high effectiveness (low regret) while improving computational efficiency over existing large-scale dueling bandit algorithms.
• To ensure theoretical soundness and practical robustness in identifying the optimal ranker despite noisy user feedback.

Proposed method

• MergeDTS uses a divide-and-conquer strategy to partition the set of rankers into small, manageable batches, limiting global comparisons.
• Within each batch, Thompson Sampling (TS) is applied to select ranker pairs for interleaving, leveraging posterior distributions to prioritize informative comparisons.
• The algorithm dynamically eliminates suboptimal rankers within batches using a UCB-based elimination strategy, reducing unnecessary comparisons.
• It integrates Double Thompson Sampling (DTS) principles into a hierarchical batching framework, improving efficiency over uniform random selection used in prior methods.
• The method is theoretically sound when the time horizon $T$ is known and the exploration parameter $\alpha > 0.5$, ensuring convergence to the optimal ranker.
• Parameter tuning is guided by empirical evaluation, with $\alpha = 0.8^6 \approx 0.2621$, $M = 16$, and $C = 4,000,000$ selected as optimal trade-offs between regret and failure rate.

Experimental results

Research questions

• RQ1 Can a scalable dueling bandit algorithm effectively identify the optimal ranker in large-scale online evaluation with thousands of rankers?
• RQ2 How does combining Thompson Sampling with batched, localized comparisons improve regret and time complexity compared to existing large-scale dueling bandit methods?
• RQ3 To what extent does MergeDTS maintain performance and robustness when the Condorcet winner is not clearly defined or when cycles exist in pairwise preferences?
• RQ4 How do the parameters $\alpha$, $M$, and $C$ affect the trade-off between regret, failure rate, and computational efficiency in practice?

Key findings

• MergeDTS reduces cumulative regret to approximately one-third of that of DTS when $\alpha = 0.8^4$, demonstrating superior effectiveness in large-scale settings.
• With $\alpha = 0.8^6 \approx 0.2621$, MergeDTS achieves a balance between low regret and minimal failure rate, with only one failure observed in 100,000 trials.
• Increasing the batch size $M$ from smaller values to $M = 16$ significantly reduces regret, as larger batches provide more informative local comparisons.
• The time complexity of MergeDTS is $O(TM^2)$, which remains tractable for realistic batch sizes, making it more efficient than DTS in large-scale scenarios.
• MergeDTS outperforms the state-of-the-art large-scale dueling bandit algorithm, MergeRUCB, in both regret and computational efficiency across multiple MSLR-Navigational benchmarks.
• Even when parameters fall outside the theoretical regime, MergeDTS maintains strong performance, indicating a practical gap between theory and empirical results.