[Paper Review] Multi-dueling Bandits with Dependent Arms

Yanan Sui, Vincent Zhuang+2 more|arXiv (Cornell University)|Apr 29, 2017

Advanced Bandit Algorithms Research3 citations

TL;DR

This paper introduces the SelfSparring algorithm for multi-dueling bandits with dependent arms, reducing the problem to a conventional bandit setting via a stochastic bandit approach like Thompson Sampling while modeling arm dependencies using a Gaussian process prior. It achieves significantly lower regret than state-of-the-art dueling bandit methods, especially as the number of arms increases, with theoretical no-regret guarantees and empirical validation across synthetic and real-world datasets.

ABSTRACT

The dueling bandits problem is an online learning framework for learning from pairwise preference feedback, and is particularly well-suited for modeling settings that elicit subjective or implicit human feedback. In this paper, we study the problem of multi-dueling bandits with dependent arms, which extends the original dueling bandits setting by simultaneously dueling multiple arms as well as modeling dependencies between arms. These extensions capture key characteristics found in many real-world applications, and allow for the opportunity to develop significantly more efficient algorithms than were possible in the original setting. We propose the \selfsparring algorithm, which reduces the multi-dueling bandits problem to a conventional bandit setting that can be solved using a stochastic bandit algorithm such as Thompson Sampling, and can naturally model dependencies using a Gaussian process prior. We present a no-regret analysis for multi-dueling setting, and demonstrate the effectiveness of our algorithm empirically on a wide range of simulation settings.

Motivation & Objective

• To address the limitations of traditional dueling bandits, which only allow pairwise comparisons and treat arms independently, by enabling simultaneous dueling of multiple arms.
• To model dependencies between arms in high-dimensional or infinite action spaces using a low-dimensional kernel structure.
• To develop a principled, efficient algorithm that reduces multi-dueling bandits to standard bandit problems while maintaining theoretical regret guarantees.
• To empirically demonstrate that the proposed method outperforms existing dueling bandit algorithms in terms of regret, particularly in large or structured action spaces.

Proposed method

• Proposes SelfSparring, an algorithm that reduces the multi-dueling bandit problem to a standard multi-armed bandit problem by treating each arm as a candidate for selection in a stochastic bandit framework.
• Uses a Gaussian process prior with a kernel function to model dependencies between arms, enabling efficient generalization across similar actions.
• Applies Thompson Sampling or similar stochastic bandit algorithms to the reduced problem to balance exploration and exploitation.
• Employs a self-sparring mechanism inspired by Sparring (Ailon et al., 2014), but with theoretical no-regret analysis, unlike the original.
• Introduces a kernelized variant, KernelSelfSparring, which models the preference function directly using a GP over the action space.
• Uses squared exponential kernels and logit link functions to generate preference probabilities from utility functions in synthetic experiments.

Experimental results

Research questions

• RQ1 Can a multi-dueling bandit framework that supports simultaneous comparisons of multiple arms achieve lower regret than pairwise dueling bandits?
• RQ2 How can dependencies between arms be effectively modeled in a dueling bandit setting to improve sample efficiency?
• RQ3 Does reducing the multi-dueling problem to a standard bandit problem with a GP prior yield better regret performance than existing dueling bandit algorithms?
• RQ4 How does the performance of the proposed method scale with increasing numbers of arms, especially in high-dimensional or continuous action spaces?

Key findings

• SelfSparring achieves no-regret performance in the multi-dueling setting, with regret growing sublinearly in time, unlike many prior dueling bandit algorithms that scale linearly with the number of arms.
• In simulations with 16-arm subsets from the MSLR-30K dataset, SelfSparring significantly outperformed the MDB algorithm in terms of cumulative regret.
• KernelSelfSparring demonstrated superior performance compared to BOPPER and GP-Sparring in kernelized 2-dueling and multi-dueling settings, particularly on the Forrester and Six-Hump Camel objective functions.
• The regret of SelfSparring did not increase substantially with the number of arms, indicating strong scalability and robustness in large action spaces.
• Empirical results show that the relative performance gain of SelfSparring over conventional dueling bandits increases with the number of arms being compared.