[Paper Review] Partially Lazy Gradient Descent for Smoothed Online Learning

Naram Mhaisen, George Iosifidis|arXiv (Cornell University)|Jan 22, 2026

Advanced Bandit Algorithms Research0 citations

TL;DR

Introduces k-lazyGD, an online learning algorithm that interpolates between greedy Online Gradient Descent and LazyGD, and proves it achieves optimal dynamic regret in Smoothed Online Convex Optimization while controlling switching costs via an FTRL-based analysis and an ensemble approach.

ABSTRACT

We introduce $k$-lazyGD, an online learning algorithm that bridges the gap between greedy Online Gradient Descent (OGD, for $k=1$) and lazy GD/dual-averaging (for $k=T$), creating a spectrum between reactive and stable updates. We analyze this spectrum in Smoothed Online Convex Optimization (SOCO), where the learner incurs both hitting and movement costs. Our main contribution is establishing that laziness is possible without sacrificing hitting performance: we prove that $k$-lazyGD achieves the optimal dynamic regret $\mathcal{O}(\sqrt{(P_T+1)T})$ for any laziness slack $k$ up to $Θ(\sqrt{T/P_T})$, where $P_T$ is the comparator path length. This result formally connects the allowable laziness to the comparator's shifts, showing that $k$-lazyGD can retain the inherently small movements of lazy methods without compromising tracking ability. We base our analysis on the Follow the Regularized Leader (FTRL) framework, and derive a matching lower bound. Since the slack depends on $P_T$, an ensemble of learners with various slacks is used, yielding a method that is provably stable when it can be, and agile when it must be.

Motivation & Objective

• Motivate and define Smoothed Online Convex Optimization (SOCO) and its hitting plus movement costs,
• Bridge the gap between greedy GD and lazy GD through a tunable laziness parameter k, forming a spectrum of update rules,
• Show that partial laziness can achieve optimal dynamic regret without sacrificing hitting performance, under a principled framework

Proposed method

• Formulate k-lazyGD within the Follow the Regularized Leader (FTRL) framework using a pruning-based gradient history mechanism
• Introduce a phased accumulation of gradients within k-length phases, with pruning governed by a counter n_t and a normal-cone-based analysis
• Derive a universal lower bound showing laziness up to k* = Theta(sqrt(T/P_T)) can preserve optimal dynamic regret
• Provide a matching upper bound and an ensemble meta-learning scheme (over multiple k and sigma) to achieve adaptivity to unknown comparator path length P_T
• Utilize an FTRL reduction with a nonstandard g_t^I term to capture pruning effects and prove equivalence to the k-lazyGD updates
• Establish staleness and stability properties of lazy iterates and connect them to improved switching cost and dynamic regret tradeoffs

Figure 1: Switching in example ( $i$ ,​ top), showing staleness , and example ( $ii$ ,​ bottom), showing stability . Left: switching cost. Right: Snapshots over 4 (top) and 2 (bottom) rounds: greedy updates move continuously, whereas lazy updates remain still or move minimally.

Experimental results

Research questions

• RQ1 What is the maximum level of laziness (k) that can be tolerated without sacrificing optimal dynamic regret in SOCO?
• RQ2 Can partially lazy updates (k-lazyGD) reproduce the hitting performance of greedy updates while retaining the stability benefits of lazy updates?
• RQ3 How does the laziness level relate to the comparator path length P_T and horizon T?
• RQ4 Can an ensemble/meta-learning approach yield adaptivity to unknown P_T while preserving order-optimal regret?
• RQ5 What are the foundational theoretical guarantees (lower and upper bounds) for k-lazyGD within the FTRL framework?

Key findings

• k-lazyGD yields dynamic regret of order O(sqrt((P_T+1)T)) for laziness up to k* = Theta(sqrt(T/P_T))
• There is a universal lower bound showing that any k-lazyGD variant incurs linear dynamic regret if laziness is too large, motivating the identified threshold
• The authors cast k-lazyGD as an instance of FTRL with a pruning rule and prove the update equivalence to the proposed k-lazyGD iteration
• An ensemble of k-lazyGD experts over a grid of sigma and k achieves adaptivity and maintains the same order-optimal regret bound across comparators
• The analysis formalizes iterate staleness and iterate stability, demonstrating how larger k reduces movement without hurting hitting performance under SOCO