[Paper Review] Online Learning in Case of Unbounded Losses Using the Follow Perturbed Leader Algorithm

Vladimir V. V’yugin|arXiv (Cornell University)|Aug 25, 2010

Advanced Bandit Algorithms Research19 references3 citations

TL;DR

This paper introduces a modified Follow the Perturbed Leader (FPL) algorithm for online learning with unbounded one-step losses, using adaptive weights based on past expert losses. It establishes optimal performance guarantees under a new notion of scaled fluctuation, proving Hannan consistency when fluctuations decay to zero, even without prior bounds on losses.

ABSTRACT

In this paper the sequential prediction problem with expert advice is considered for the case where losses of experts suffered at each step cannot be bounded in advance. We present some modification of Kalai and Vempala algorithm of following the perturbed leader where weights depend on past losses of the experts. New notions of a volume and a scaled fluctuation of a game are introduced. We present a probabilistic algorithm protected from unrestrictedly large one-step losses. This algorithm has the optimal performance in the case when the scaled fluctuations of one-step losses of experts of the pool tend to zero.

Motivation & Objective

• Address the gap in online learning theory where expert losses are unbounded, a scenario often excluded in standard expert advice frameworks.
• Develop a robust learning algorithm that remains effective even when individual losses grow without prior bounds.
• Introduce new game-theoretic concepts—volume and scaled fluctuation—to characterize the complexity of sequential prediction games with unbounded losses.
• Achieve optimal performance guarantees (Hannan consistency) under minimal assumptions, specifically when the scaled fluctuation of expert losses tends to zero.
• Provide a probabilistic algorithm with adaptive learning rates that ensures expected cumulative loss close to the best expert in hindsight, even under unbounded loss growth.

Proposed method

• Modify the classical FPL algorithm by introducing weights dependent on historical cumulative losses of experts, replacing fixed perturbations.
• Define the scaled fluctuation of a game as a measure of loss variability relative to a time-varying scale function, enabling analysis under unbounded losses.
• Introduce the concept of 'volume' of a game to quantify the effective complexity of the prediction problem in terms of expert loss dynamics.
• Use a probabilistic perturbation mechanism with exponential i.i.d. noise, but adapt the learning rate dynamically based on observed loss patterns.
• Apply Kolmogorov’s three-series theorem and Kronecker’s lemma to prove almost-sure convergence of normalized loss differences, ensuring long-term consistency.
• Design the algorithm PROT (Perturbed Regularized Tracker) to dynamically adjust to decreasing scaled fluctuations, achieving optimal regret bounds.

Experimental results

Research questions

• RQ1 Can a follow-the-perturbed-leader algorithm maintain optimal regret bounds when one-step losses are unbounded and not a priori constrained?
• RQ2 What game-theoretic measures can characterize the difficulty of online prediction under unbounded losses, beyond boundedness assumptions?
• RQ3 Is Hannan consistency achievable in the presence of unbounded losses if the scaled fluctuation of expert losses tends to zero?
• RQ4 Can an adaptive learning rate strategy ensure optimal performance without prior knowledge of loss bounds or growth rates?
• RQ5 What is the role of the volume and scaled fluctuation of a game in determining the convergence and performance of online learning algorithms?

Key findings

• The proposed algorithm achieves an expected cumulative loss bound of $ E(s_{1:t}) \ leq \min_i s^i_{1:t} + \frac{\log N}{\epsilon} $, with $ \epsilon $ adaptively chosen based on observed loss fluctuations.
• When the scaled fluctuation $ \text{fluc}(t) \to 0 $, the algorithm is Hannan consistent, meaning the learner’s average loss converges to that of the best expert in hindsight.
• The algorithm is proven to be asymptotically consistent in the mean for games where $ \limsup_{t\to\infty} \frac{\text{fluc}(t)}{\gamma_i(t)} < \infty $ for some $ i $, even when $ \gamma_i(t) $ is not known in advance.
• The use of adaptive weights and dynamic learning rates allows the algorithm to handle loss growth rates faster than polynomial but slower than exponential.
• The theoretical framework introduces the volume of a game and scaled fluctuation as key metrics, enabling performance analysis beyond bounded loss assumptions.
• The proof relies on Kolmogorov’s three-series theorem and Kronecker’s lemma to establish almost-sure convergence of normalized loss differences, ensuring long-term stability and consistency.