[Paper Review] Online Learning for Changing Environments using Coin Betting

Kwang-Sung Jun, Francesco Orabona+2 more|arXiv (Cornell University)|Nov 6, 2017

Advanced Bandit Algorithms Research4 references3 citations

TL;DR

This paper proposes a novel meta-algorithm, Coin Betting for Changing Environments (CBCE), that achieves a strongly-adaptive regret bound of $\sqrt{(I_2 - I_1)\log(I_2)}$, which is $\sqrt{\log T}$ better than prior methods with the same time complexity. It leverages a parameter-free reduction from the coin betting problem to handle sleeping experts, enabling first-order strongly-adaptive regret bounds and outperforming state-of-the-art methods in both expert advice and metric learning tasks.

ABSTRACT

A key challenge in online learning is that classical algorithms can be slow to adapt to changing environments. Recent studies have proposed "meta" algorithms that convert any online learning algorithm to one that is adaptive to changing environments, where the adaptivity is analyzed in a quantity called the strongly-adaptive regret. This paper describes a new meta algorithm that has a strongly-adaptive regret bound that is a factor of $\sqrt{\log(T)}$ better than other algorithms with the same time complexity, where $T$ is the time horizon. We also extend our algorithm to achieve a first-order (i.e., dependent on the observed losses) strongly-adaptive regret bound for the first time, to our knowledge. At its heart is a new parameter-free algorithm for the learning with expert advice (LEA) problem in which experts sometimes do not output advice for consecutive time steps (i.e., \emph{sleeping} experts). This algorithm is derived by a reduction from optimal algorithms for the so-called coin betting problem. Empirical results show that our algorithm outperforms state-of-the-art methods in both learning with expert advice and metric learning scenarios.

Motivation & Objective

• To address the challenge of slow adaptation in online learning under non-stationary environments.
• To develop a meta-algorithm that achieves tighter strongly-adaptive regret bounds than existing methods.
• To extend the framework to support first-order regret bounds, depending on observed losses rather than time horizon.
• To provide a parameter-free solution for the sleeping experts problem via reduction to the coin betting problem.
• To empirically validate superior performance in expert advice and metric learning scenarios.

Proposed method

• The core method is a reduction from the learning with expert advice (LEA) problem with sleeping experts to the coin betting problem, enabling parameter-free optimization.
• The algorithm uses a geometric covering (GC) or data streaming (DS) interval decomposition to manage time intervals and bound regret across all subintervals.
• A novel regret decomposition technique is applied to derive the strongly-adaptive regret bound using the structure of interval overlaps and cumulative loss tracking.
• The method incorporates a parameter-free update rule derived from optimal coin betting strategies, which dynamically adjusts to loss patterns without tuning.
• The algorithm achieves a time complexity factor of $\log T$, matching the best-known meta-algorithms, while improving the regret bound by a $\sqrt{\log T}$ factor.
• Theoretical analysis proves that the SA-Regret is bounded by $O\left(\sqrt{(I_2 - I_1)\log(I_2)}\right)$, improving over prior $O\left(\sqrt{(I_2 - I_1)\log^2(I_2)}\right)$ bounds.

Experimental results

Research questions

• RQ1 Can a meta-algorithm be designed to achieve a strongly-adaptive regret bound that is $\sqrt{\log T}$ better than existing methods with the same time complexity?
• RQ2 Is it possible to derive a first-order strongly-adaptive regret bound that depends on observed losses rather than the time horizon?
• RQ3 Can a parameter-free algorithm for sleeping experts be constructed via reduction to the coin betting problem?
• RQ4 How does the proposed algorithm perform in practice compared to state-of-the-art methods in dynamic environments?
• RQ5 What are the theoretical and practical implications of using geometric covering versus data streaming interval decompositions?

Key findings

• The proposed CBCE algorithm achieves a strongly-adaptive regret bound of $O\left(\sqrt{(I_2 - I_1)\log(I_2)}\right)$, which is $\sqrt{\log T}$ better than prior methods with the same time complexity.
• The algorithm is the first to achieve a first-order strongly-adaptive regret bound, depending on observed losses rather than the time horizon $T$.
• Empirical results show that CBCE outperforms state-of-the-art methods in both expert advice and metric learning tasks under changing environments.
• The reduction from sleeping experts to coin betting enables a parameter-free algorithm that adapts without tuning hyperparameters.
• Theoretical analysis confirms that the algorithm maintains strong regret guarantees under both geometric covering and data streaming interval decompositions, with minor constant factor differences.
• The method is robust to black-box algorithms with anytime regret bounds, and can be adapted to fixed-budget algorithms via the doubling trick.