[Paper Review] A Modern Introduction to Online Learning

Francesco Orabona|arXiv (Cornell University)|Dec 31, 2019

Advanced Bandit Algorithms Research92 references68 citations

TL;DR

A modern, comprehensive text on Online Convex Optimization, detailing online learning algorithms (FTL, OGD, OMD, FTRL) with regret analyses, adaptive methods, and foundational connections to broader topics.

ABSTRACT

In this monograph, I introduce the basic concepts of Online Learning through a modern view of Online Convex Optimization. Here, online learning refers to the framework of regret minimization under worst-case assumptions. I present first-order and second-order algorithms for online learning with convex losses, in Euclidean and non-Euclidean settings. All the algorithms are clearly presented as instantiation of Online Mirror Descent or Follow-The-Regularized-Leader and their variants. Particular attention is given to the issue of tuning the parameters of the algorithms and learning in unbounded domains, through adaptive and parameter-free online learning algorithms. Non-convex losses are dealt through convex surrogate losses and through randomization. The bandit setting is also briefly discussed, touching on the problem of adversarial and stochastic multi-armed bandits. These notes do not require prior knowledge of convex analysis and all the required mathematical tools are rigorously explained. Moreover, all the included proofs have been carefully chosen to be as simple and as short as possible.

Motivation & Objective

• Introduce the online learning framework and regret minimization as a core objective.
• Present key online algorithms (e.g., Online Gradient Descent, Subgradient Descent, Mirror Descent, FTRL) and their regret analyses.
• Explore extensions: adaptive/parameter-free methods, strong convexity, bandits, and non-stationary settings.
• Relate online learning to broader topics such as convex analysis, stochastic optimization, and learning theory.

Proposed method

• Define regret with respect to an arbitrary comparator and adversarial loss sequences.
• Develop and analyze core algorithms: Online Subgradient Descent, Online Gradient Descent with projection, and Follow-the-Regularized-Leader variants.
• Introduce Online Mirror Descent and its connections to subgradients and Bregman divergence.
• Provide proofs and lemmas (e.g., Be-the-Leader) to establish sublinear regret under convexity and bounded gradients.

Experimental results

Research questions

• RQ1 What regret bounds can be guaranteed for online learning with convex (differentiable or non-differentiable) losses?
• RQ2 How do Online Gradient Descent, Subgradient Descent, and Mirror Descent compare under various convexity and domain conditions?
• RQ3 What roles do adaptivity and parameter-free approaches play in achieving sublinear regret in unbounded or complex domains?
• RQ4 How can online learning concepts extend to advanced settings such as bandits, saddle-point problems, and sequential investments?

Key findings

• FTL can be suboptimal in adversarial settings, motivating Online Gradient Descent with projection.
• Projected Online Gradient Descent yields sublinear regret bounds under convex differentiable losses and bounded domains.
• Be-the-Leader lemma underpins sublinear regret by relating cumulative losses of adaptive leaders to a fixed comparator.
• Online Mirror Descent provides a unifying view via Bregman divergences and can handle non-Euclidean geometries.
• Adaptive and parameter-free variants, as well as strong convexity, improve performance and broaden applicability.