[Paper Review] Efficient Methods for Non-stationary Online Learning

Peng Zhao, Yanfeng Xie+2 more|arXiv (Cornell University)|Sep 16, 2023

Advanced Bandit Algorithms ResearchDecision Sciences3 citations

TL;DR

This paper proposes efficient online learning algorithms for non-stationary environments that reduce computational cost by minimizing projections to the feasible set from O(log T) to just one per round. By leveraging a novel reduction mechanism from parameter-free learning, the method achieves optimal dynamic and adaptive regret bounds with only one gradient query and one function evaluation per round, significantly improving efficiency over existing two-layer ensemble approaches.

ABSTRACT

Non-stationary online learning has drawn much attention in recent years. In particular, dynamic regret and adaptive regret are proposed as two principled performance measures for online convex optimization in non-stationary environments. To optimize them, a two-layer online ensemble is usually deployed due to the inherent uncertainty of non-stationarity, in which multiple base-learners are maintained and a meta-algorithm is employed to track the best one on the fly. However, the two-layer structure raises concerns about computational complexity -- such methods typically maintain $O(\log T)$ base-learners simultaneously for a $T$-round online game and thus perform multiple projections onto the feasible domain per round, which becomes the computational bottleneck when the domain is complicated. In this paper, we present efficient methods for optimizing dynamic regret and adaptive regret that reduce the number of projections per round from $O(\log T)$ to $1$. The proposed algorithms require only one gradient query and one function evaluation at each round. Our technique hinges on the reduction mechanism developed in parameter-free online learning and requires non-trivial modifications for non-stationary online methods. Furthermore, we study an even stronger measure, namely "interval dynamic regret", and reduce the number of projections per round from $O(\log^2 T)$ to $1$ for minimizing it. Our reduction demonstrates broad generality and applies to two important applications: online stochastic control and online principal component analysis, resulting in methods that are both efficient and optimal. Finally, empirical studies verify our theoretical findings.

Motivation & Objective

• To address the high computational cost of existing two-layer online ensemble methods for non-stationary online learning.
• To reduce the number of projections per round from O(log T) to 1 while maintaining optimal regret guarantees.
• To develop efficient algorithms for both dynamic regret and adaptive regret in online convex optimization.
• To achieve optimal regret bounds with minimal computational overhead—only one gradient query and one function evaluation per round.
• To extend parameter-free learning techniques to non-stationary settings through non-trivial algorithmic adaptations.

Proposed method

• The method employs a reduction mechanism from parameter-free online learning to design a single-projection algorithm for non-stationary online convex optimization.
• It uses time-varying regularizers and a carefully tuned learning rate schedule to balance regret and stability.
• The algorithm maintains only one active learner per round, eliminating the need for maintaining O(log T) base-learners as in traditional two-layer ensembles.
• A key innovation lies in adapting the self-confident learning rate tuning from parameter-free learning to the non-stationary setting.
• The approach ensures that the regret bound scales optimally with path length PT and function variation, matching known theoretical limits.
• Theoretical analysis relies on novel lemmas for handling time-varying regularizers and bounding cumulative gradients.

Experimental results

Research questions

• RQ1 Can we design online learning algorithms for non-stationary environments that require only one projection per round?
• RQ2 Is it possible to achieve optimal dynamic and adaptive regret bounds with minimal computational overhead?
• RQ3 How can parameter-free learning techniques be adapted to non-stationary online convex optimization?
• RQ4 What is the minimal number of gradient queries and function evaluations needed to maintain optimal regret in non-stationary settings?
• RQ5 Can the two-layer ensemble structure be replaced with a more efficient single-learner framework without sacrificing regret performance?

Key findings

• The proposed algorithm reduces the number of projections per round from O(log T) to 1, significantly lowering computational cost in complex feasible domains.
• The method achieves a dynamic regret bound of O(√(T(1 + PT))) for general convex functions, matching the best-known theoretical guarantees.
• For smooth functions, the algorithm attains a small-loss dynamic regret bound of O(√((FT + PT)(1 + PT))), improving upon prior work.
• The algorithm requires only one gradient query and one function evaluation per round, making it highly efficient.
• Empirical evaluations confirm that the theoretical performance gains are realized in practice, with strong regret performance and low computational cost.
• The reduction technique from parameter-free learning enables a clean, unified framework that avoids the complexity of maintaining multiple base-learners.