[Paper Review] A Near-Optimal Change-Detection Based Algorithm for Piecewise-Stationary Combinatorial Semi-Bandits

Huozhi Zhou, Lingda Wang+2 more|arXiv (Cornell University)|Aug 27, 2019

Advanced Bandit Algorithms Research39 references4 citations

TL;DR

This paper proposes GLR-CUCB, a near-optimal algorithm for piecewise-stationary combinatorial semi-bandits that combines the CUCB algorithm with a generalized likelihood ratio (GLR) change-point detector to adaptively detect distribution shifts. The method achieves a regret upper bound of $\mathcal{O}(\sqrt{NKT\log T})$, nearly matching the derived $\Omega(\sqrt{NKT})$ minimax lower bound, and outperforms state-of-the-art baselines on synthetic and real-world datasets.

ABSTRACT

We investigate the piecewise-stationary combinatorial semi-bandit problem. Compared to the original combinatorial semi-bandit problem, our setting assumes the reward distributions of base arms may change in a piecewise-stationary manner at unknown time steps. We propose an algorithm, exttt{GLR-CUCB}, which incorporates an efficient combinatorial semi-bandit algorithm, exttt{CUCB}, with an almost parameter-free change-point detector, the \emph{Generalized Likelihood Ratio Test} (GLRT). Our analysis shows that the regret of exttt{GLR-CUCB} is upper bounded by $\mathcal{O}(\sqrt{NKT\log{T}})$, where $N$ is the number of piecewise-stationary segments, $K$ is the number of base arms, and $T$ is the number of time steps. As a complement, we also derive a nearly matching regret lower bound on the order of $Ω(\sqrt{NKT}$), for both piecewise-stationary multi-armed bandits and combinatorial semi-bandits, using information-theoretic techniques and judiciously constructed piecewise-stationary bandit instances. Our lower bound is tighter than the best available regret lower bound, which is $Ω(\sqrt{T})$. Numerical experiments on both synthetic and real-world datasets demonstrate the superiority of exttt{GLR-CUCB} compared to other state-of-the-art algorithms.

Motivation & Objective

• To address the gap in combinatorial multi-armed bandit literature for non-stationary environments where reward distributions change abruptly at unknown times.
• To develop an efficient, adaptive algorithm that detects distribution shifts in combinatorial semi-bandits without prior knowledge of change points or distribution parameters.
• To establish a tight regret lower bound for piecewise-stationary combinatorial semi-bandits, showing the fundamental difficulty of the problem.
• To empirically validate the superiority of the proposed algorithm over existing state-of-the-art methods in both synthetic and real-world settings.

Proposed method

• The algorithm integrates the CUCB (Combinatorial Upper Confidence Bound) framework with a generalized likelihood ratio (GLR) test as a change-point detector to monitor base arm reward distributions.
• The GLR detector identifies changes in the mean reward of base arms without requiring prior knowledge of the change magnitude or distribution parameters.
• Upon detecting a change, the algorithm performs a global restart of the CUCB procedure to re-estimate the optimal superarm under the new distribution.
• The method is designed to be almost parameter-free, relying only on a small number of hyperparameters such as the significance level $\delta$ and a confidence threshold $p$.
• Theoretical analysis bounds the regret of GLR-CUCB by $\mathcal{O}(\sqrt{NKT\log T})$, where $N$ is the number of segments, $K$ the number of base arms, and $T$ the time horizon.
• An information-theoretic lower bound of $\Omega(\sqrt{NKT})$ is derived for both piecewise-stationary MAB and CMAB, demonstrating near-optimality of the proposed algorithm.

Experimental results

Research questions

• RQ1 Can an efficient, parameter-free change-detection mechanism be effectively integrated into combinatorial semi-bandit algorithms to handle non-stationary reward distributions?
• RQ2 What is the fundamental limit (minimax regret) of piecewise-stationary combinatorial semi-bandits, and how close can an algorithm get to it?
• RQ3 Does the proposed GLR-CUCB algorithm achieve near-optimal regret in practice, especially when the change-point detection is not perfectly aligned with actual distribution shifts?
• RQ4 How does the performance of GLR-CUCB compare to oracle and state-of-the-art algorithms in non-stationary environments with unknown and irregular changes?

Key findings

• GLR-CUCB achieves a regret upper bound of $\mathcal{O}(\sqrt{NKT\log T})$, which is nearly order-optimal, as the minimax lower bound is $\Omega(\sqrt{NKT})$.
• The derived minimax regret lower bound is tighter than the previously known $\Omega(\sqrt{T})$ bound, highlighting the role of $N$ and $K$ in the problem complexity.
• On synthetic data with $T=5000$, $K=6$, $m=2$, and $N=5$, GLR-CUCB performs nearly as well as Oracle-CUCB and significantly better than other algorithms, even when Assumption 4.1 (on segment length) is violated.
• On the Yahoo! dataset with $T=22500$, $K=6$, $m=2$, and $N=9$, GLR-CUCB outperforms all baselines and shows only a small gap from Oracle-CUCB, demonstrating robustness in real-world non-stationary settings.
• Surprisingly, the variant LR-GLR-CUCB outperforms Oracle-CUCB in some cases, suggesting that global restarts may be suboptimal when changes are small and do not affect the optimal superarm.
• The experiments confirm that change-point detection with GLR is effective even under mild violations of theoretical assumptions, indicating strong practical robustness.