[Paper Review] Bridging Adversarial and Nonstationary Multi-armed Bandit

Ningyuan Chen, Shuoguang Yang+1 more|arXiv (Cornell University)|Jan 5, 2022

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper introduces a unified multi-armed bandit framework that bridges adversarial and nonstationary bandits via a switch budget $S_T$, which controls the number of times the optimal action sequence can change. The proposed AE3 algorithm achieves near-optimal regret that transitions smoothly between adversarial ($\tilde{O}(\sqrt{KS_T T})$) and nonstationary ($\tilde{O}((KV_T)^{1/3}T^{2/3})$) regimes based on $S_T$ and variational budget $V_T$, with phase transition at $S_T \asymp K^{-1/3}V_T^{2/3}T^{1/3}$.

ABSTRACT

In the multi-armed bandit framework, there are two formulations that are commonly employed to handle time-varying reward distributions: adversarial bandit and nonstationary bandit. Although their oracles, algorithms, and regret analysis differ significantly, we provide a unified formulation in this paper that smoothly bridges the two as special cases. The formulation uses an oracle that takes the best-fixed arm within time windows. Depending on the window size, it turns into the oracle in hindsight in the adversarial bandit and dynamic oracle in the nonstationary bandit. We provide algorithms that attain the optimal regret with the matching lower bound.

Motivation & Objective

• Address the gap between adversarial and nonstationary bandit frameworks, which differ in oracles, algorithms, and regret bounds.
• Unify these two settings under a single formulation using a switch budget $S_T$ that controls the number of allowed action switches.
• Develop an algorithm that adapts to both adversarial and nonstationary environments by dynamically adjusting to $S_T$ and $V_T$.
• Establish tight regret bounds that reflect a phase transition at $S_T \asymp K^{-1/3}V_T^{2/3}T^{1/3}$, capturing the trade-off between conservativeness and adaptability.
• Provide theoretical and empirical validation of the algorithm's optimality and robustness across varying levels of nonstationarity and switch constraints.

Proposed method

• The unified oracle is defined as the maximum cumulative reward over all action sequences with at most $S_T$ switches, interpolating between the static oracle ($S_T = 1$) and dynamic oracle ($S_T \geq T$).
• An adaptive exploration strategy is designed to balance exploitation of current estimates and exploration of potential switch points, using a time-dependent confidence bound.
• The AE3 algorithm partitions the time horizon into blocks of size $\Delta = \lceil T / S_T \rceil$ when $S_T$ is small, and applies a modified EXP3-style update within each block.
• Regret analysis is conducted in two regimes: when $S_T \geq K^{-1/3}V_T^{2/3}T^{1/3}$, the regret scales as $\tilde{O}((KV_T)^{1/3}T^{2/3})$; when $S_T \leq K^{-1/3}V_T^{2/3}T^{1/3}$, it scales as $\tilde{O}(\sqrt{KS_T T})$.
• Lower bounds are derived to show that the regret bounds are tight up to logarithmic factors, confirming the optimality of the AE3 algorithm.
• Empirical validation is performed via simulations across varying $T$, $K$, $S_T$, and $V_T$, with log-log plots confirming theoretical slopes of $2/3$, $1/3$, and $1/2$ for $T$, $V_T$, and $K$ respectively.

Experimental results

Research questions

• RQ1 How can adversarial and nonstationary bandit problems be formally unified under a single framework with a continuous parameter?
• RQ2 What is the optimal regret achievable when the agent's expectation is constrained by a switch budget $S_T$ in a nonstationary environment?
• RQ3 Does the regret exhibit a phase transition at a critical switch budget $S_T \asymp K^{-1/3}V_T^{2/3}T^{1/3}$, and if so, how does it manifest in the bounds?
• RQ4 Can an algorithm achieve near-optimal regret across both adversarial and nonstationary regimes without prior knowledge of $S_T$?
• RQ5 Is the proposed AE3 algorithm optimal in both regimes, and do empirical results match the theoretical predictions?

Key findings

• When the switch budget $S_T$ is large ($S_T \geq K^{-1/3}V_T^{2/3}T^{1/3}$), the regret is bounded by $\tilde{O}((KV_T)^{1/3}T^{2/3})$, matching the nonstationary bandit lower bound.
• When the switch budget $S_T$ is small ($S_T \leq K^{-1/3}V_T^{2/3}T^{1/3}$), the regret is bounded by $\tilde{O}(\sqrt{KS_T T})$, matching the adversarial bandit lower bound.
• The regret exhibits a phase transition at $S_T \asymp K^{-1/3}V_T^{2/3}T^{1/3}$, where the dominant scaling shifts from dependence on $S_T$ to $V_T$.
• Empirical results confirm the theoretical slopes: $\log$-regret vs. $\log T$ has slope $2/3$, vs. $\log V_T$ has slope $1/3$, and vs. $\log K$ has slope $1/3$ or $1/2$ depending on the regime.
• The number of arms $K$ affects regret as $K^{1/3}$ in the large-$S_T$ regime and $K^{1/2}$ in the small-$S_T$ regime, consistent with theoretical bounds.
• AE3 achieves near-optimality in both regimes, with empirical performance closely matching theoretical predictions across multiple simulation settings.