[Paper Review] Online Optimization : Competing with Dynamic Comparators

Ali Jadbabaie, Alexander Rakhlin+2 more|arXiv (Cornell University)|Jan 26, 2015

Advanced Bandit Algorithms Research13 references93 citations

TL;DR

This paper introduces a fully adaptive online optimization algorithm that achieves dynamic regret bounds scaling with three complexity measures: the path variation of the comparator sequence ($C_T$), the temporal variability of loss functions ($V_T$), and the prediction error of gradients ($D_T$). By leveraging an optimistic mirror descent framework with adaptive step-sizes, the method achieves sublinear regret without prior knowledge of these quantities, improving upon existing bounds in both static and dynamic regret settings.

ABSTRACT

Recent literature on online learning has focused on developing adaptive algorithms that take advantage of a regularity of the sequence of observations, yet retain worst-case performance guarantees. A complementary direction is to develop prediction methods that perform well against complex benchmarks. In this paper, we address these two directions together. We present a fully adaptive method that competes with dynamic benchmarks in which regret guarantee scales with regularity of the sequence of cost functions and comparators. Notably, the regret bound adapts to the smaller complexity measure in the problem environment. Finally, we apply our results to drifting zero-sum, two-player games where both players achieve no regret guarantees against best sequences of actions in hindsight.

Motivation & Objective

• To develop an online learning algorithm that adapts to both the regularity of the comparator sequence and the niceness of nature’s loss functions.
• To unify existing regret bounds that depend on $C_T$, $V_T$, and $D_T$ into a single framework without requiring prior knowledge of these measures.
• To establish sublinear regret guarantees in the full information setting by combining dynamic regret with adaptive step-sizes and optimistic predictions.
• To extend the applicability of online optimization to non-i.i.d. and non-adversarial environments by exploiting temporal structure in loss functions.
• To demonstrate the effectiveness of the method in drifting two-player zero-sum games, where both players achieve no-regret against time-varying optimal strategies.

Proposed method

• The algorithm uses an optimistic mirror descent (OMD) framework with adaptive step-sizes to balance exploration and exploitation in dynamic environments.
• It incorporates a prediction mechanism for gradients using a sequence $\hat{f}_{t-1}$, enabling regret bounds dependent on $D_T = \sum_t \|\nabla f_t(x_t) - M_t\|_*^2$.
• The regret analysis leverages telescoping sums and norm inequalities to bound the difference between actual and predicted losses, particularly using $\ell_1$ and $\ell_\infty$ norms.
• A key component is the use of a step-size schedule $\eta_t$ that depends on $\log(T^2n)$ and $L$, ensuring convergence even when $V_T$ is unknown.
• The method derives bounds on $\sum_t \|f_t^\top A_t - f_{t-1}^\top A_{t-1}\|_\infty^2$, which captures temporal variation in payoff matrices.
• It establishes regret bounds that scale with $C_T(u)$, $V_T$, and $D_T$, combining them through a unified analysis that adapts to the smallest complexity measure.

Experimental results

Research questions

• RQ1 Can an online algorithm achieve dynamic regret that adapts to the path variation of the comparator sequence $C_T$ without prior knowledge of its value?
• RQ2 How can temporal variability $V_T$ of loss functions be exploited to improve regret bounds in online convex optimization?
• RQ3 Can a single algorithm simultaneously achieve regret bounds dependent on multiple complexity measures ($C_T$, $V_T$, $D_T$) in a fully adaptive manner?
• RQ4 What is the interplay between optimistic prediction and regret minimization in non-i.i.d. environments with drifting cost functions?
• RQ5 In two-player zero-sum games, can both players achieve no-regret against time-varying optimal strategies using this method?

Key findings

• The proposed algorithm achieves a dynamic regret bound of order $\mathcal{O}(\log(T^2n)(C_T + 2)(32L + o(1)))$ without prior knowledge of $C_T$.
• The regret bound scales with $\sqrt{\sum_t \|f_t^\top A_t - f_{t-1}^\top A_{t-1}\|_\infty^2}$, capturing temporal variation in payoff matrices.
• When $V_T$ is small, the regret bound improves significantly, achieving $\mathcal{O}(T^{2/3}(V_T + 1)^{1/3})$ under noisy gradients, matching known results but without requiring $V_T$ to be known in advance.
• The method achieves sublinear regret in drifting two-player zero-sum games, with both players converging to the average minimax equilibrium at a rate dependent on $C_T$ and $V_T$.
• The analysis shows that the regret bound adapts to the smallest of the three complexity measures: $C_T$, $V_T$, and $D_T$, providing a unified improvement over prior work.
• The algorithm’s performance is robust even when one player is dishonest, as the regret bound remains sublinear and depends only on the opponent’s strategy variation and the learner’s own prediction error.