[Paper Review] Merge Double Thompson Sampling for Large Scale Online Ranker Evaluation

Chang Li, Ilya Markov+2 more|arXiv (Cornell University)|Dec 11, 2018

Advanced Bandit Algorithms Research3 citations

TL;DR

This paper proposes Merge Double Thompson Sampling (MergeDTS), a scalable K-armed dueling bandit algorithm for large-scale online ranker evaluation under the Condorcet assumption. By combining divide-and-conquer with Thompson Sampling, MergeDTS reduces regret and time complexity, outperforming state-of-the-art dueling bandit methods in large-scale web search evaluation settings.

ABSTRACT

Online ranker evaluation is one of the key challenges in information retrieval. While the preferences of rankers can be inferred by interleaved comparison methods, how to effectively choose the pair of rankers to generate the result list without degrading the user experience too much can be formalized as a K-armed dueling bandit problem, which is an online partial-information learning framework, where feedback comes in the form of pair-wise preferences. A commercial search system may evaluate a large number of rankers concurrently, and scaling effectively in the presence of numerous rankers has not been fully studied. In this paper, we focus on solving the large-scale online ranker evaluation problem under the so-called Condorcet assumption, where there exists an optimal ranker that is preferred to all other rankers. We propose Merge Double Thompson Sampling (MergeDTS), which first utilizes a divide-and-conquer strategy that localizes the comparisons carried out by the algorithm to small batches of rankers, and then employs the Thompson Sampling (TS) to reduce the comparisons between suboptimal rankers inside these small batches. The effectiveness (regret) and efficiency (time complexity) of MergeDTS are extensively evaluated using examples from the domain of online evaluation for web search. Our main finding is that for large-scale Condorcet ranker evaluation problems MergeDTS outperforms the state-of-the-art dueling bandit algorithms.

Motivation & Objective

• Address the challenge of efficiently evaluating a large number of rankers in online information retrieval systems.
• Scale online ranker evaluation effectively in the presence of numerous rankers while maintaining user experience.
• Formalize the problem as a K-armed dueling bandit under the Condorcet assumption, where an optimal ranker exists that dominates all others.
• Develop a method that minimizes regret and computational cost in large-scale settings through localized comparison and adaptive sampling.
• Achieve superior performance compared to existing dueling bandit algorithms in terms of regret and time complexity.

Proposed method

• Apply a divide-and-conquer strategy to partition the large set of rankers into smaller, manageable batches.
• Within each batch, use Thompson Sampling to adaptively select and compare rankers based on posterior belief over preference outcomes.
• Limit comparisons to suboptimal rankers by leveraging probabilistic sampling, reducing unnecessary evaluations.
• Merge results across batches using a hierarchical aggregation strategy to identify the globally optimal ranker.
• Maintain low regret by focusing exploration on promising rankers while minimizing comparisons with clearly inferior ones.
• Ensure scalability by restricting comparisons to localized batches, reducing the overall time complexity of the algorithm.

Experimental results

Research questions

• RQ1 Can a divide-and-conquer approach effectively scale dueling bandit algorithms to large-scale online ranker evaluation?
• RQ2 How does Thompson Sampling reduce regret within localized batches of rankers compared to alternative selection strategies?
• RQ3 To what extent does MergeDTS reduce time complexity while maintaining low regret in large-scale Condorcet settings?
• RQ4 How does MergeDTS compare in performance to state-of-the-art dueling bandit algorithms in real-world web search evaluation scenarios?
• RQ5 What is the impact of batch size and merging strategy on the overall regret and convergence speed of the algorithm?

Key findings

• MergeDTS significantly reduces regret compared to existing dueling bandit algorithms in large-scale online ranker evaluation.
• The algorithm achieves lower time complexity by localizing comparisons within small batches of rankers.
• Thompson Sampling within batches effectively reduces comparisons involving suboptimal rankers, improving efficiency.
• The divide-and-conquer strategy enables scalable performance even when evaluating hundreds or thousands of rankers.
• Empirical evaluation on web search examples confirms that MergeDTS outperforms state-of-the-art methods in both regret and computational efficiency.
• The method maintains strong performance under the Condorcet assumption, where a single optimal ranker exists.