[Paper Review] Contextual Combinatorial Conservative Bandits

Xiaojin Zhang, Shuai Li+2 more|arXiv (Cornell University)|Nov 26, 2019

Advanced Bandit Algorithms Research35 references4 citations

TL;DR

This paper introduces a novel framework for contextual combinatorial conservative bandits, where agents select multiple arms at each round while ensuring instantaneous reward safety. It proposes two UCB-based algorithms—one with known conservative reward and one with unknown—achieving a regret bound of $\tilde{O}(d^2 + d\sqrt{T})$, with tighter bounds in high-dimensional settings than prior work.

ABSTRACT

The problem of multi-armed bandits (MAB) asks to make sequential decisions while balancing between exploitation and exploration, and have been successfully applied to a wide range of practical scenarios. Various algorithms have been designed to achieve a high reward in a long term. However, its short-term performance might be rather low, which is injurious in risk sensitive applications. Building on previous work of conservative bandits, we bring up a framework of contextual combinatorial conservative bandits. An algorithm is presented and a regret bound of $ ilde O(d^2+d\sqrt{T})$ is proven, where $d$ is the dimension of the feature vectors, and $T$ is the total number of time steps. We further provide an algorithm as well as regret analysis for the case when the conservative reward is unknown. Experiments are conducted, and the results validate the effectiveness of our algorithm.

Motivation & Objective

• To address the lack of instantaneous safety guarantees in combinatorial and contextual bandit settings, especially in risk-sensitive applications.
• To extend the conservative bandit framework to the contextual combinatorial setting, where multiple arms are selected and rewards are non-linear.
• To design computationally efficient algorithms that maintain safety at every time step while maximizing long-term reward.
• To provide tighter regret bounds in high-dimensional feature spaces compared to prior conservative bandit methods.
• To handle both cases where the conservative reward threshold is known and unknown, ensuring robustness in practical deployment.

Proposed method

• Proposes a contextual combinatorial conservative bandit framework where the agent selects up to $K$ arms per round based on contextual features, with semi-bandit feedback.
• Designs a UCB-based algorithm (CCConUCB) that maintains a conservative reward constraint by setting a lower bound on expected cumulative reward per round.
• Uses a non-linear reward function $f(A,w)$ for arm set $A$ and context $w$, satisfying two mild assumptions (e.g., boundedness and smoothness).
• Incorporates a conservative policy that ensures the selected arm set's expected reward is at least $1-\alpha$ of the conservative baseline, even in early rounds.
• Employs a novel regret analysis technique that derives a tighter bound of $\tilde{O}(d^2 + d\sqrt{T})$ for high-dimensional contexts, improving upon prior $O(d\sqrt{T}\log T + (d\log d)^2)$ bounds.
• Adapts the algorithm for unknown conservative reward by estimating the baseline reward from historical data, maintaining safety through confidence intervals.

Experimental results

Research questions

• RQ1 Can a contextual combinatorial bandit algorithm maintain instantaneous reward safety while maximizing long-term cumulative reward?
• RQ2 How can the conservative bandit principle be extended to combinatorial action sets with non-linear reward functions?
• RQ3 What is the theoretical regret performance of a conservative algorithm in high-dimensional contextual settings?
• RQ4 How does the algorithm's performance compare to non-conservative counterparts in terms of constraint violations and regret?
• RQ5 How does the conservative coefficient $\alpha$ affect the convergence speed and the time to outperform a purely conservative policy?

Key findings

• The proposed CCConUCB algorithm successfully maintains the cumulative reward above $1-\alpha$ of the conservative baseline at every time step, even in early rounds.
• The algorithm achieves a regret bound of $\tilde{O}(d^2 + d\sqrt{T})$, which is tighter than the $O(d\sqrt{T}\log T + (d\log d)^2)$ bound from prior work in high-dimensional settings.
• When reduced to the standard contextual bandit ($K=1$), the regret bound becomes $O(d\sqrt{T} + d^{3/2})$, improving upon the $O(d\sqrt{T}\log T + (d\log d)^2)$ result from [15].
• Numerical simulations show that CCConUCB achieves lower regret than non-conservative UCB while avoiding constraint violations, especially when $\alpha$ is small.
• The algorithm converges faster and outperforms the purely conservative policy more quickly when $\alpha$ is larger, due to increased exploration opportunities.
• With smaller $\alpha$, the agent explores less and degrades in regret more slowly, but still avoids large constraint violations, demonstrating strong safety guarantees.