[Paper Review] Combinatorial Multi-Objective Multi-Armed Bandit Problem

Doruk Öner, Altuğ Karakurt+2 more|arXiv (Cornell University)|Mar 11, 2018

Advanced Bandit Algorithms Research14 references3 citations

TL;DR

This paper introduces the Combinatorial Multi-Objective Multi-Armed Bandit (COMO-MAB) problem, which unifies combinatorial and multi-objective online learning by modeling actions as sets of arms whose rewards are linear combinations of individual arm rewards. It proposes COMO-UCB, a fair UCB-based algorithm achieving $O(NL^3\log T)$ Pareto regret, outperforming existing methods in multi-user communication and recommender systems by balancing exploration, exploitation, and fairness across objectives.

ABSTRACT

In this paper, we introduce the COmbinatorial Multi-Objective Multi-Armed Bandit (COMO-MAB) problem that captures the challenges of combinatorial and multi-objective online learning simultaneously. In this setting, the goal of the learner is to choose an action at each time, whose reward vector is a linear combination of the reward vectors of the arms in the action, to learn the set of super Pareto optimal actions, which includes the Pareto optimal actions and actions that become Pareto optimal after adding an arbitrary small positive number to their expected reward vectors. We define the Pareto regret performance metric and propose a fair learning algorithm whose Pareto regret is $O(N L^3 \log T)$, where $T$ is the time horizon, $N$ is the number of arms and $L$ is the maximum number of arms in an action. We show that COMO-MAB has a wide range of applications, including recommending bundles of items to users and network routing, and focus on a resource-allocation application for multi-user communication in the presence of multidimensional performance metrics, where we show that our algorithm outperforms existing MAB algorithms.

Motivation & Objective

• To address the lack of unified frameworks for combinatorial and multi-objective online learning in multi-armed bandit settings.
• To define a new performance metric, Pareto regret, that captures cumulative loss due to suboptimal action selection in multi-objective contexts.
• To design a fair learning algorithm that selects actions from the super Pareto front uniformly at random to balance performance across all objectives.
• To demonstrate the effectiveness of the proposed algorithm in real-world applications such as multi-user communication, recommender systems, and network routing.
• To show that COMO-UCB achieves significantly lower regret and higher fairness compared to state-of-the-art MAB algorithms in multi-objective settings.

Proposed method

• Introduces the COMO-MAB problem where actions are sets of arms and rewards are linear combinations of individual arm rewards across multiple objectives.
• Defines the super Pareto front (SPF) to include actions that become Pareto optimal with an infinitesimal reward increase, addressing zero suboptimality gaps in the standard Pareto front.
• Proposes COMO-UCB, a UCB-based algorithm that maintains upper confidence bounds for each arm’s reward vector and selects actions from the estimated SPF uniformly at random.
• Employs a multi-dimensional Hoeffding’s inequality to bound the deviation of reward estimates and defines a multi-dimensional notion of suboptimality for regret analysis.
• Uses a subset of the SPF that dominates the selected action the most to evaluate regret, enabling tighter bounds on cumulative loss.
• Applies technical innovations in multi-objective regret analysis, including novel confidence intervals and dominance comparisons across reward vectors.

Experimental results

Research questions

• RQ1 How can combinatorial and multi-objective learning be jointly modeled in a multi-armed bandit framework to address real-world applications with multiple conflicting performance metrics?
• RQ2 What is an appropriate performance metric for multi-objective combinatorial bandits that accounts for actions with zero suboptimality gap but not in the Pareto front?
• RQ3 Can a fair learning algorithm be designed that uniformly samples from the super Pareto front to balance performance across all objectives?
• RQ4 What regret bounds can be proven for a UCB-based algorithm in this new COMO-MAB setting, and how do they scale with the number of arms and action size?
• RQ5 How does the proposed COMO-UCB algorithm compare in performance and fairness to existing MAB algorithms in practical multi-objective scenarios?

Key findings

• COMO-UCB achieves $O(NL^3\log T)$ Pareto regret, where $N$ is the number of arms and $L$ is the maximum number of arms per action, proving theoretical efficiency in multi-objective combinatorial learning.
• In the multi-user communication setup with $T = 10^5$ time steps, COMO-UCB selects an action from the Pareto front $79\%$ of the time, significantly outperforming SO-UCB1 ($37\%$) and Pareto UCB1 ($48\%$).
• COMO-UCB demonstrates superior fairness: it selects each of the 9 actions in the SPF with nearly equal frequency, unlike SO-UCB1 and LLR, which favor one action disproportionately.
• The regret of COMO-UCB grows significantly slower than competitors, as shown in simulations, confirming its faster convergence to optimal actions.
• For network routing, the expected regret of COMO-UCB is bounded by $\mathbb{E}[\mathrm{Reg}(T)] \leq \Delta_{\max}\left(\frac{4NL^2(L+1)\log(T\sqrt[4]{2})}{\Delta_{\min}^2} + N + \frac{\pi^2}{3}NL\right)$, showing scalability with path length and network size.
• COMO-UCB outperforms LLR and Pareto UCB1 in both regret and fairness, even though LLR is the closest competitor in regret, highlighting the importance of fairness in multi-objective bandits.