[Paper Review] Multi-objective Contextual Multi-armed Bandit Problem with a Dominant Objective.

Cem Tekin, Eralp Turğay|arXiv (Cornell University)|Aug 18, 2017

Advanced Bandit Algorithms Research21 references3 citations

TL;DR

This paper introduces the Contextual Multi-armed Bandit with a Dominant Objective (CMAB-DO), a multi-objective bandit framework where one objective dominates the other. The proposed MOC-MAB algorithm maximizes the non-dominant objective among arms that optimize the dominant objective, achieving sublinear 2D and Pareto regret, with empirical validation on synthetic and real-world datasets.

ABSTRACT

In this paper, we propose a new multi-objective contextual multi-armed bandit problem with two objectives, where one of the objectives dominates the other objective. Unlike single-objective bandit problems in which the learner obtains a random scalar reward for each arm it selects, in the proposed problem, the learner obtains a random reward vector, where each component of the reward vector corresponds to one of the objectives and the distribution of the reward depends on the context that is provided to the learner at the beginning of each round. We call this problem contextual multi-armed bandit with a dominant objective (CMAB-DO). In CMAB-DO, the goal of the learner is to maximize its total reward in the non-dominant objective while ensuring that it maximizes its total reward in the dominant objective. In this case, the optimal arm given a context is the one that maximizes the expected reward in the non-dominant objective among all arms that maximize the expected reward in the dominant objective. First, we show that the optimal arm lies in the Pareto front. Then, we propose the multi-objective contextual multi-armed bandit algorithm (MOC-MAB), and define two performance measures: the 2-dimensional (2D) regret and the Pareto regret. We show that both the 2D regret and the Pareto regret of MOC-MAB are sublinear in the number of rounds. We also compare the performance of the proposed algorithm with other state-of-the-art methods in synthetic and real-world datasets. The proposed model and the algorithm have a wide range of real-world applications that involve multiple and possibly conflicting objectives ranging from wireless communication to medical diagnosis and recommender systems.

Motivation & Objective

• To address multi-objective decision-making in contextual bandits where one objective dominates the other, reflecting real-world scenarios with hierarchical priorities.
• To define a new problem setting—CMAB-DO—where the learner selects arms based on context and receives a vector reward with two components: one dominant and one non-dominant.
• To formalize the optimal arm as the one maximizing the non-dominant reward among all arms that maximize the dominant reward under a given context.
• To design an algorithm that balances exploration and exploitation while ensuring dominance in the primary objective and optimization in the secondary.
• To evaluate performance using two novel regret measures: 2D regret and Pareto regret, both proven to be sublinear in the number of rounds.

Proposed method

• The CMAB-DO problem models reward vectors where each component corresponds to one of two objectives, with the dominant objective's expected reward determining the feasible set of arms.
• The optimal arm is defined as the one that maximizes the expected non-dominant reward among all arms achieving the maximum expected reward in the dominant objective.
• The proposed MOC-MAB algorithm uses context-dependent arm selection and maintains confidence bounds for both objectives to guide exploration.
• The algorithm computes upper confidence bounds for both objectives and selects arms that are optimal in the dominant objective while maximizing the non-dominant one.
• Two performance metrics are introduced: 2D regret, which measures cumulative loss in both objectives, and Pareto regret, which evaluates performance relative to the Pareto front.
• Theoretical analysis proves that both 2D regret and Pareto regret of MOC-MAB are sublinear in the number of rounds, indicating convergence to optimal performance.

Experimental results

Research questions

• RQ1 How can a multi-objective contextual bandit problem be formalized when one objective dominates the other in terms of priority?
• RQ2 What is the structure of the optimal arm in such a problem, and how does it relate to the Pareto front?
• RQ3 Can an algorithm be designed that ensures maximization of the dominant objective while optimizing the non-dominant one?
• RQ4 What performance metrics are appropriate for evaluating such a multi-objective bandit setting?
• RQ5 Do the proposed regret measures (2D and Pareto regret) remain sublinear over time, indicating effective learning?

Key findings

• The optimal arm in the CMAB-DO problem lies on the Pareto front, as it maximizes the non-dominant objective among arms that are optimal in the dominant objective.
• The MOC-MAB algorithm achieves sublinear 2D regret, indicating that the cumulative loss in both objectives grows slower than linearly with the number of rounds.
• The MOC-MAB algorithm also achieves sublinear Pareto regret, showing that the algorithm's performance converges to the Pareto optimal frontier over time.
• Empirical evaluation on synthetic and real-world datasets demonstrates that MOC-MAB outperforms state-of-the-art methods in balancing the dominant and non-dominant objectives.
• The theoretical analysis confirms that both 2D and Pareto regret are sublinear, validating the algorithm's long-term effectiveness in multi-objective decision-making under dominance constraints.