[Paper Review] A parameter-free hedging algorithm

Kamalika Chaudhuri, Yoav Freund+1 more|arXiv (Cornell University)|Mar 16, 2009

Advanced Bandit Algorithms Research20 references64 citations

TL;DR

This paper introduces NormalHedge, a parameter-free online learning algorithm for decision-theoretic online learning (DTOL) that adapts dynamically without requiring manual tuning of learning rates. It achieves regret bounds of $ Oig( esizemath{12pt}{!}{\sqrt{T\ln\frac{1}{\epsilon}} + \ln^2 N}\big) $ for the top $\epsilon$-quantile of actions, matching the performance of optimally-tuned Hedge algorithms while being robust to large action sets.

ABSTRACT

We study the problem of decision-theoretic online learning (DTOL). Motivated by practical applications, we focus on DTOL when the number of actions is very large. Previous algorithms for learning in this framework have a tunable learning rate parameter, and a barrier to using online-learning in practical applications is that it is not understood how to set this parameter optimally, particularly when the number of actions is large. In this paper, we offer a clean solution by proposing a novel and completely parameter-free algorithm for DTOL. We introduce a new notion of regret, which is more natural for applications with a large number of actions. We show that our algorithm achieves good performance with respect to this new notion of regret; in addition, it also achieves performance close to that of the best bounds achieved by previous algorithms with optimally-tuned parameters, according to previous notions of regret.

Motivation & Objective

• To address the practical challenge of tuning the learning rate in online learning algorithms when the number of actions $N$ is very large.
• To propose a novel, completely parameter-free algorithm that eliminates the need for manual hyperparameter adjustment.
• To introduce a new notion of regret—regret to the top $\epsilon$-quantile of actions—that is more natural for applications with many near-optimal actions.
• To achieve regret bounds that are competitive with optimally-tuned Hedge algorithms, even when $N$ is large.

Proposed method

• The algorithm uses a potential-based framework where each action is assigned a potential $ \phi(x,c) = \exp\big(\frac{([x]_+)^2}{2c}\big) $, with $ x $ being the action's regret and $ c $ an adaptive scale parameter.
• Action weights are updated proportionally to the derivative of their potential with respect to regret, enabling dynamic adaptation.
• The scale parameter $ c_t $ is updated online based on the loss sequence, ensuring the algorithm adapts to the observed regret growth.
• The algorithm maintains a balance between exploration and exploitation by adjusting the curvature of the potential function in response to cumulative regret.
• It uses a line search in each round to compute the optimal $ c_t $, ensuring the regret bound is maintained without prior knowledge of $ T $ or $ N $.

Experimental results

Research questions

• RQ1 Can a parameter-free online learning algorithm be designed that performs well without requiring tuning of the learning rate, especially when $ N $ is large?
• RQ2 Is there a more natural notion of regret than the standard regret to the best action in applications with many near-optimal actions?
• RQ3 Can a parameter-free algorithm achieve regret bounds competitive with optimally-tuned Hedge algorithms under this new notion of regret?
• RQ4 How can the adaptive scale parameter $ c_t $ be updated to ensure tight regret bounds without prior knowledge of $ T $ or $ N $?

Key findings

• The NormalHedge algorithm achieves a regret bound of $ O\big(\sqrt{T\ln\frac{1}{\epsilon}} + \ln^2 N\big) $ for the top $\epsilon$-quantile of actions, valid simultaneously for all $ T $ and $ \epsilon $.
• When $ \epsilon = 1/N $, the regret to the best action is bounded by $ O\big(\sqrt{T\ln N} + \ln^2 N\big) $, which is only slightly worse than the optimal $ O(\sqrt{T\ln N}) $ bound of Hedge with optimal tuning.
• The algorithm is completely parameter-free, requiring no manual setting of the learning rate $ \eta $, making it practical for large-scale applications.
• The regret bound holds uniformly across all rounds and quantile levels, and the algorithm adapts to the observed loss sequence through dynamic adjustment of the scale parameter $ c_t $.