[Paper Review] Utility-based Dueling Bandits as a Partial Monitoring Game

Pratik Gajane, Tanguy Urvoy|arXiv (Cornell University)|Jul 10, 2015

Advanced Bandit Algorithms Research12 references3 citations

TL;DR

This paper frames the utility-based dueling bandit problem as a partial monitoring game and proves it belongs to the easy category of the time-regret hierarchy, achieving a regret bound of $\tilde{\Theta}(\sqrt{T})$. It demonstrates that while general partial monitoring algorithms suffer from suboptimal $\tilde{\mathcal{O}}(K\sqrt{T})$ regret due to quadratic action space, dedicated algorithms like REX3 achieve the optimal $\tilde{\mathcal{O}}(\sqrt{KT})$ regret, highlighting the need for structure-aware algorithms.

ABSTRACT

Partial monitoring is a generic framework for sequential decision-making with incomplete feedback. It encompasses a wide class of problems such as dueling bandits, learning with expect advice, dynamic pricing, dark pools, and label efficient prediction. We study the utility-based dueling bandit problem as an instance of partial monitoring problem and prove that it fits the time-regret partial monitoring hierarchy as an easy - i.e. Theta (sqrt{T})- instance. We survey some partial monitoring algorithms and see how they could be used to solve dueling bandits efficiently. Keywords: Online learning, Dueling Bandits, Partial Monitoring, Partial Feedback, Multiarmed Bandits

Motivation & Objective

• To formalize the utility-based dueling bandit problem within the partial monitoring (PM) framework.
• To determine the complexity class of the dueling bandit problem in the PM time-regret hierarchy.
• To evaluate the performance of existing PM algorithms on dueling bandits, especially regarding dependence on the number of actions $K^2$.
• To identify the limitations of general PM algorithms in capturing structural properties of dueling bandits.
• To demonstrate the superiority of structure-aware algorithms like REX3 over general PM methods in terms of regret scaling.

Proposed method

• The authors model the dueling bandit problem as a partial monitoring game using the formalism of action set $\boldsymbol{N}$, outcome set $\boldsymbol{M}$, loss function $\mathcal{L}$, gain function $\mathcal{G}$, and feedback function $\mathcal{H}$.
• They prove the problem is locally observable by showing no matrix $\mathcal{B}$ exists that maps feedback to gain, implying it is not globally observable.
• They apply the PM hierarchy from Bartók et al. (2014) to classify the dueling bandit problem as an 'easy' instance with $\tilde{\Theta}(\sqrt{T})$ regret.
• They analyze existing PM algorithms (e.g., FEEDEXP3, BALATON, CBP, GLOBAL-EXP3, SAVAGE, Neighborhood Watch) and compare their regret bounds in the context of dueling bandits.
• They show that general PM algorithms incur $\tilde{\mathcal{O}}(K\sqrt{T})$ regret due to $N \approx K^2$ actions, while REX3 achieves $\tilde{\mathcal{O}}(\sqrt{KT})$ by exploiting structure.
• They use contradiction to prove the impossibility of a linear feedback-to-gain mapping, justifying the need for specialized algorithms.

Experimental results

Research questions

• RQ1 Is the utility-based dueling bandit problem an instance of the partial monitoring framework, and if so, how is it formally structured?
• RQ2 In the PM time-regret hierarchy, to which complexity class does the dueling bandit problem belong?
• RQ3 Why do general PM algorithms perform poorly on dueling bandits despite theoretical guarantees?
• RQ4 Can the structure of the dueling bandit problem be exploited to achieve better regret bounds than general PM algorithms?
• RQ5 What is the optimal regret scaling achievable for dueling bandits, and which algorithm achieves it?

Key findings

• The utility-based dueling bandit problem is classified as an 'easy' instance in the PM time-regret hierarchy, with a regret bound of $\tilde{\Theta}(\sqrt{T})$.
• The problem is not globally observable, as no linear feedback-to-gain mapping $\mathcal{B}$ exists, which invalidates certain general PM algorithm assumptions.
• General PM algorithms like FEEDEXP3, BALATON, CBP, GLOBAL-EXP3, SAVAGE, and Neighborhood Watch all achieve $\tilde{\mathcal{O}}(K\sqrt{T})$ regret due to the $K^2$-sized action space.
• Among existing PM algorithms, only REX3 achieves the optimal $\tilde{\mathcal{O}}(\sqrt{KT})$ regret bound, which is sublinear in $K$ and optimal for dueling bandits.
• The counterexample for GLOBAL-EXP3 shows $N' \approx K^2$ in the point-local game, leading to $\tilde{\mathcal{O}}(K\sqrt{T})$ regret, confirming its suboptimality for dueling bandits.
• The study establishes that structure-aware algorithms are essential for achieving optimal regret in dueling bandit problems, as general PM methods fail to exploit the underlying structure.