[Paper Review] Adaptive Online Learning in Dynamic Environments

Lijun Zhang, Shiyin Lu+1 more|arXiv (Cornell University)|Oct 25, 2018

Advanced Bandit Algorithms Research14 references52 citations

TL;DR

The paper introduces Ader, an adaptive online learning method that achieves the optimal dynamic regret bound for general dynamic environments, with an improved version that reduces gradient evaluations to 1 per round and extensions to dynamical models.

ABSTRACT

In this paper, we study online convex optimization in dynamic environments, and aim to bound the dynamic regret with respect to any sequence of comparators. Existing work have shown that online gradient descent enjoys an $O(\sqrt{T}(1+P_T))$ dynamic regret, where $T$ is the number of iterations and $P_T$ is the path-length of the comparator sequence. However, this result is unsatisfactory, as there exists a large gap from the $Ω(\sqrt{T(1+P_T)})$ lower bound established in our paper. To address this limitation, we develop a novel online method, namely adaptive learning for dynamic environment (Ader), which achieves an optimal $O(\sqrt{T(1+P_T)})$ dynamic regret. The basic idea is to maintain a set of experts, each attaining an optimal dynamic regret for a specific path-length, and combines them with an expert-tracking algorithm. Furthermore, we propose an improved Ader based on the surrogate loss, and in this way the number of gradient evaluations per round is reduced from $O(\log T)$ to $1$. Finally, we extend Ader to the setting that a sequence of dynamical models is available to characterize the comparators.

Motivation & Objective

• Motivate robust online learning under changing environments by bounding dynamic regret against arbitrary comparator sequences.
• Establish a tight lower bound for general dynamic regret and develop an algorithm achieving the matching upper bound.
• Develop an adaptive framework that combines multiple experts with different path-length optimals.
• Reduce gradient query complexity via a surrogate loss based variant without sacrificing performance.
• Extend the approach to settings with a sequence of dynamical models governing the comparators.

Proposed method

• Maintain a set of experts, each optimal for a specific path-length, and run them in parallel.
• Use a meta-algorithm (exponentially weighted) to track the best expert at each round.
• Expert algorithms are online gradient descent instances with different step sizes.
• Introduce a surrogate loss to reduce gradient evaluations from O(log T) to 1 per round in the improved version.
• Extend Ader to incorporate a sequence of dynamical models and analyze regret in terms of P_T′.
• Provide proofs showing an overall dynamic regret bound of O(sqrt(T(1+P_T))).

Experimental results

Research questions

• RQ1 What is the minimax lower bound for general dynamic regret in online convex optimization?
• RQ2 Can we design an algorithm that achieves the optimal O(sqrt(T(1+P_T))) dynamic regret adaptively for any comparator sequence?
• RQ3 How can the number of gradient evaluations be reduced without sacrificing regret performance?
• RQ4 Can the framework accommodate a sequence of dynamical models and yield tight bounds in terms of P_T′?
• RQ5 What extensions are feasible to handle dynamical models while preserving optimal regret?

Key findings

• Ader achieves the optimal O(sqrt(T(1+P_T))) dynamic regret for general dynamic environments.
• A lower bound of Omega(sqrt(T(1+P_T))) is established for the general dynamic regret.
• An improved Ader using surrogate loss reduces gradient evaluations per round from O(log T) to 1.
• The regret bound remains O(sqrt(T(1+P_T))) when using the surrogate loss variant.
• Extensions to contraction-based dynamical models yield O(sqrt(T(1+P_T′))) dynamic regret, improving prior results.
• The bounds are adaptive to the path-length of the comparator sequence, shrinking when comparators move slowly.