[Paper Review] Revisiting Smoothed Online Learning

Lijun Zhang, Wei Jiang+2 more|arXiv (Cornell University)|Feb 13, 2021

Advanced Bandit Algorithms Research58 references4 citations

TL;DR

This paper revisits smoothed online learning by proposing a simple yet effective approach that balances hitting and switching costs through optimization. It achieves improved competitive ratios—up to $1 + \frac{2}{\sqrt{\lambda}}$—for convex, $\lambda$-quadratic growth functions, and introduces Smoothed Ader, an algorithm that attains an optimal $O(\sqrt{T(1+P_T)})$ dynamic regret bound with switching cost, confirmed by a matching lower bound.

ABSTRACT

In this paper, we revisit the problem of smoothed online learning, in which the online learner suffers both a hitting cost and a switching cost, and target two performance metrics: competitive ratio and dynamic regret with switching cost. To bound the competitive ratio, we assume the hitting cost is known to the learner in each round, and investigate the simple idea of balancing the two costs by an optimization problem. Surprisingly, we find that minimizing the hitting cost alone is $\max(1, \frac{2}α)$-competitive for $α$-polyhedral functions and $1 + \frac{4}λ$-competitive for $λ$-quadratic growth functions, both of which improve state-of-the-art results significantly. Moreover, when the hitting cost is both convex and $λ$-quadratic growth, we reduce the competitive ratio to $1 + \frac{2}{\sqrtλ}$ by minimizing the weighted sum of the hitting cost and the switching cost. To bound the dynamic regret with switching cost, we follow the standard setting of online convex optimization, in which the hitting cost is convex but hidden from the learner before making predictions. We modify Ader, an existing algorithm designed for dynamic regret, slightly to take into account the switching cost when measuring the performance. The proposed algorithm, named as Smoothed Ader, attains an optimal $O(\sqrt{T(1+P_T)})$ bound for dynamic regret with switching cost, where $P_T$ is the path-length of the comparator sequence. Furthermore, if the hitting cost is accessible in the beginning of each round, we obtain a similar guarantee without the bounded gradient condition, and establish an $Ω(\sqrt{T(1+P_T)})$ lower bound to confirm the optimality.

Motivation & Objective

• To improve competitive ratio bounds in smoothed online learning by analyzing the performance of minimizing hitting cost alone and weighted sum minimization.
• To develop a dynamic regret bound with switching cost that adapts to path-length regularity of comparator sequences.
• To establish optimality of the proposed algorithm through a matching $\Omega(\sqrt{T(1+P_T)})$ lower bound.
• To unify insights between competitive ratio and dynamic regret by analyzing different assumptions on hitting cost functions.
• To provide theoretical foundations for switch-constrained learning and continual learning through smoothed online learning.

Proposed method

• Proposes minimizing the weighted sum of hitting cost and switching cost at each round as a greedy strategy to balance costs.
• Analyzes competitive ratio under $\alpha$-polyhedral and $\lambda$-quadratic growth functions, deriving bounds of $\max(1, \frac{2}{\alpha})$ and $1 + \frac{4}{\lambda}$, respectively.
• Introduces Smoothed Ader (SAder), a modified version of Ader that incorporates switching cost into performance evaluation.
• Derives an $O(\sqrt{T(1+P_T)})$ dynamic regret bound for SAder under standard online convex optimization with hidden hitting costs.
• Develops a lookahead variant of SAder that uses prior knowledge of hitting costs to achieve the same bound without bounded gradient assumptions.
• Establishes a matching $\Omega(\sqrt{T(1+P_T)})$ lower bound to confirm optimality of the dynamic regret bound.

Experimental results

Research questions

• RQ1 Can minimizing the hitting cost alone achieve better competitive ratios than existing methods for $\alpha$-polyhedral and $\lambda$-quadratic growth functions?
• RQ2 What competitive ratio can be achieved by a greedy algorithm that minimizes the weighted sum of hitting and switching costs for convex, $\lambda$-quadratic growth functions?
• RQ3 Can dynamic regret with switching cost be bounded optimally using a modified Ader algorithm that accounts for switching costs?
• RQ4 Does the proposed Smoothed Ader algorithm achieve the optimal $O(\sqrt{T(1+P_T)})$ dynamic regret bound under path-length regularity?
• RQ5 Is the $O(\sqrt{T(1+P_T)})$ bound tight, as confirmed by a matching $\Omega(\sqrt{T(1+P_T)})$ lower bound?

Key findings

• Minimizing the hitting cost alone achieves a $\max(1, \frac{2}{\alpha})$-competitive ratio for $\alpha$-polyhedral functions, improving upon prior results.
• For $\lambda$-quadratic growth functions, minimizing the hitting cost alone yields a $1 + \frac{4}{\lambda}$-competitive ratio, which is a significant improvement over state-of-the-art.
• When the hitting cost is both convex and $\lambda$-quadratic growth, minimizing the weighted sum of hitting and switching costs achieves a $1 + \frac{2}{\sqrt{\lambda}}$-competitive ratio.
• The proposed Smoothed Ader (SAder) algorithm achieves an $O(\sqrt{T(1+P_T)})$ dynamic regret bound with switching cost, matching the theoretical lower bound.
• A lookahead variant of SAder achieves the same bound without requiring bounded gradients, confirming the robustness of the result.
• An $\Omega(\sqrt{T(1+P_T)})$ lower bound is established, proving that the $O(\sqrt{T(1+P_T)})$ dynamic regret bound is optimal.