[Paper Review] Contextual Combinatorial Bandits with Probabilistically Triggered Arms

Xutong Liu, Jinhang Zuo+5 more|arXiv (Cornell University)|Mar 30, 2023

Advanced Bandit Algorithms ResearchDecision Sciences3 citations

TL;DR

This paper introduces Contextual Combinatorial Bandits with Probabilistically Triggered Arms (C²MAB-T), proposing the C²-UCB-T and VAC²-UCB algorithms to address regret minimization under smoothness conditions. It achieves an $ tilde{O}(d\sqrt{KT})$ regret bound under the triggering probability modulated (TPM) condition and an improved $ tilde{O}(d\sqrt{T})$ bound under variance modulated (VM) or TPVM conditions, removing dependency on $K$ and $1/p_{\min}$, with empirical validation on synthetic and real-world data.

ABSTRACT

We study contextual combinatorial bandits with probabilistically triggered arms (C$^2$MAB-T) under a variety of smoothness conditions that capture a wide range of applications, such as contextual cascading bandits and contextual influence maximization bandits. Under the triggering probability modulated (TPM) condition, we devise the C$^2$-UCB-T algorithm and propose a novel analysis that achieves an $ ilde{O}(d\sqrt{KT})$ regret bound, removing a potentially exponentially large factor $O(1/p_{\min})$, where $d$ is the dimension of contexts, $p_{\min}$ is the minimum positive probability that any arm can be triggered, and batch-size $K$ is the maximum number of arms that can be triggered per round. Under the variance modulated (VM) or triggering probability and variance modulated (TPVM) conditions, we propose a new variance-adaptive algorithm VAC$^2$-UCB and derive a regret bound $ ilde{O}(d\sqrt{T})$, which is independent of the batch-size $K$. As a valuable by-product, our analysis technique and variance-adaptive algorithm can be applied to the CMAB-T and C$^2$MAB setting, improving existing results there as well. We also include experiments that demonstrate the improved performance of our algorithms compared with benchmark algorithms on synthetic and real-world datasets.

Motivation & Objective

• To formalize a general framework for contextual combinatorial bandits with probabilistically triggered arms (C²MAB-T) to model diverse applications like cascading bandits and influence maximization.
• To address the limitations of prior work that either assume deterministic feedback or suffer from suboptimal regret bounds involving $1/p_{\min}$ or $K$.
• To develop algorithms that achieve regret bounds independent of $K$ under realistic smoothness conditions such as variance modulation.
• To provide a refined analysis that removes the $O(1/p_{\min})$ factor present in prior results, improving scalability and performance.

Proposed method

• Proposes the C²-UCB-T algorithm under the triggering probability modulated (TPM) condition, using a novel analysis to achieve $\tilde{O}(d\sqrt{KT})$ regret without the $1/p_{\min}$ factor.
• Introduces the variance-adaptive VAC²-UCB algorithm under the variance modulated (VM) or triggering probability and variance modulated (TPVM) conditions, achieving $\tilde{O}(d\sqrt{T})$ regret independent of $K$.
• Employs a new confidence bound design that adapts to the variance of arm outcomes, enabling tighter regret analysis.
• Uses a time-varying feature map $\bm{\phi}_t$ to model contextual information and assumes linear mean rewards in $\theta^*$, ensuring scalability.
• Applies a probabilistic arm-triggering process to generalize feedback models, including semi-bandit, cascading, and probabilistic feedback.
• Derives regret bounds through a refined analysis of the expected reward gap and arm sampling counts, leveraging smoothness conditions on the reward function.

Experimental results

Research questions

• RQ1 Can we design a contextual combinatorial bandit framework that generalizes existing models like cascading bandits and influence maximization while maintaining scalability?
• RQ2 Does it remain possible to achieve regret bounds independent of the batch-size $K$ and the minimum triggering probability $p_{\min}$ under realistic smoothness assumptions?
• RQ3 Can a variance-adaptive algorithm improve regret performance in settings with probabilistically triggered arms compared to prior conservative exploration strategies?
• RQ4 How does the proposed analysis technique eliminate the $O(1/p_{\min})$ factor that plagues previous approaches in combinatorial bandits?
• RQ5 To what extent can the new algorithmic and analytical framework be extended to improve existing results in CMAB-T and C²MAB?

Key findings

• The C²-UCB-T algorithm achieves a regret bound of $\tilde{O}(d\sqrt{KT})$ under the TPM condition, removing the $O(1/p_{\min})$ factor present in prior work.
• The VAC²-UCB algorithm achieves a regret bound of $\tilde{O}(d\sqrt{T})$ under the VM or TPVM conditions, which is independent of the batch-size $K$.
• The analysis technique and variance-adaptive design are transferable to CMAB-T and C²MAB, improving existing regret bounds in those settings.
• Empirical results on synthetic and real-world datasets demonstrate superior performance of the proposed algorithms compared to benchmark methods.
• The improved regret bounds are achieved through a tighter analysis of the expected reward gap and adaptive confidence intervals that scale with variance.
• The framework successfully generalizes to applications such as contextual cascading bandits and influence maximization bandits, with provable guarantees.