[Paper Review] Efficient-UCBV: An Almost Optimal Algorithm using Variance Estimates

Subhojyoti Mukherjee, K. P. Naveen+2 more|arXiv (Cornell University)|Nov 9, 2017

Advanced Bandit Algorithms Research16 citations

TL;DR

This paper proposes EUCBV, an efficient UCB-based algorithm for stochastic multi-armed bandits that integrates variance estimates and arm elimination to achieve near-optimal regret bounds. By combining variance-aware confidence bounds with aggressive exploration and non-uniform arm selection, EUCBV achieves a gap-independent regret of $O(\sqrt{KT})$ and a gap-dependent regret of $O\left(\frac{K\sigma^{2}_{\text{max}}\log(T\Delta^{2}/K)}{\Delta}\right)$, outperforming UCB1, UCBV, MOSS, and OCUCB in extensive experiments.

ABSTRACT

We propose a novel variant of the UCB algorithm (referred to as Efficient-UCB-Variance (EUCBV)) for minimizing cumulative regret in the stochastic multi-armed bandit (MAB) setting. EUCBV incorporates the arm elimination strategy proposed in UCB-Improved \citep{auer2010ucb}, while taking into account the variance estimates to compute the arms' confidence bounds, similar to UCBV \citep{audibert2009exploration}. Through a theoretical analysis we establish that EUCBV incurs a \emph{gap-dependent} regret bound of {\scriptsize $O\left( \dfrac{Kσ^2_{\max} \log (TΔ^2 /K)}Δ ight)$} after $T$ trials, where $Δ$ is the minimal gap between optimal and sub-optimal arms; the above bound is an improvement over that of existing state-of-the-art UCB algorithms (such as UCB1, UCB-Improved, UCBV, MOSS). Further, EUCBV incurs a \emph{gap-independent} regret bound of {\scriptsize $O\left(\sqrt{KT} ight)$} which is an improvement over that of UCB1, UCBV and UCB-Improved, while being comparable with that of MOSS and OCUCB. Through an extensive numerical study we show that EUCBV significantly outperforms the popular UCB variants (like MOSS, OCUCB, etc.) as well as Thompson sampling and Bayes-UCB algorithms.

Motivation & Objective

• To develop a bandit algorithm that minimizes cumulative regret in stochastic multi-armed bandit problems.
• To improve upon existing UCB variants by incorporating variance estimates into confidence bounds.
• To achieve both gap-dependent and gap-independent regret bounds that are order-optimal or near-optimal.
• To outperform state-of-the-art algorithms like MOSS, OCUCB, and Thompson sampling in diverse experimental environments.
• To design a stable, variance-aware algorithm that performs well over large horizons and with high-variance optimal arms.

Proposed method

• EUCBV uses variance estimates to compute tighter confidence bounds for each arm, similar to UCBV.
• It applies the arm elimination strategy from UCB-Improved, removing sub-optimal arms after rounds of exploration.
• The algorithm employs non-uniform arm selection based on variance and confidence intervals to prioritize arms with higher uncertainty and better potential reward.
• It combines aggressive exploration parameters with variance-aware confidence intervals to accelerate sub-optimal arm elimination.
• The confidence bound incorporates both the empirical mean and the empirical variance of each arm’s rewards.
• The algorithm requires the horizon $T$ as input, and is designed to achieve optimal regret when $T \geq K^{2.4}$.

Experimental results

Research questions

• RQ1 Can variance estimates be effectively integrated into UCB-style algorithms to improve regret bounds?
• RQ2 Does combining arm elimination with variance-aware confidence bounds lead to better empirical performance than existing state-of-the-art algorithms?
• RQ3 Can a UCB variant achieve both order-optimal gap-independent regret and improved gap-dependent regret compared to UCB1 and UCBV?
• RQ4 How does EUCBV perform in environments with small gaps and high-variance optimal arms, where Bayesian methods may fail?
• RQ5 Is EUCBV more stable and scalable than Thompson sampling and Bayes-UCB in large-horizon, high-arms settings?

Key findings

• EUCBV achieves a gap-independent regret bound of $O(\sqrt{KT})$, matching the best-known bound of MOSS and OCUCB.
• Its gap-dependent regret bound is $O\left(\frac{K\sigma^{2}_{\text{max}}\log(T\Delta^{2}/K)}{\Delta}\right)$, which improves upon UCB1, UCBV, and UCB-Improved.
• In experiments with 100 arms and high-variance optimal arms, EUCBV significantly outperformed TS-G, BU-G, UCBV, MOSS, and OCUCB.
• EUCBV outperformed variance-aware algorithms like UCBV due to its aggressive exploration and non-uniform arm selection strategy.
• In a uniform-gap setting with three variance groups, EUCBV showed much slower regret growth than TS-G, MOSS, BU-G, OCUCB, and UCBV.
• Bayesian algorithms like TS-G and BU-G failed in high-variance optimal arm scenarios, while EUCBV remained robust and effective.