[Paper Review] On the optimality of the Hedge algorithm in the stochastic regime

Jaouad Mourtada, Stéphane Gaïffas|arXiv (Cornell University)|Sep 5, 2018

Advanced Bandit Algorithms Research14 citations

TL;DR

This paper demonstrates that the anytime Hedge algorithm with a decreasing learning rate $η_t \propto \sqrt{\log M / t}$ achieves optimal $O(\log M / \Delta)$ regret in the stochastic regime—matching the best adaptive algorithms—without requiring specialized tuning. Despite its simplicity and non-adaptive learning rate, it adapts to easier stochastic environments, outperforming fixed-horizon and doubling-trick variants that fail to adapt.

ABSTRACT

In this paper, we study the behavior of the Hedge algorithm in the online stochastic setting. We prove that anytime Hedge with decreasing learning rate, which is one of the simplest algorithm for the problem of prediction with expert advice, is surprisingly both worst-case optimal and adaptive to the easier stochastic and adversarial with a gap problems. This shows that, in spite of its small, non-adaptive learning rate, Hedge possesses the same optimal regret guarantee in the stochastic case as recently introduced adaptive algorithms. Moreover, our analysis exhibits qualitative differences with other variants of the Hedge algorithm, such as the fixed-horizon version (with constant learning rate) and the one based on the so-called "doubling trick", both of which fail to adapt to the easier stochastic setting. Finally, we discuss the limitations of anytime Hedge and the improvements provided by second-order regret bounds in the stochastic case.

Motivation & Objective

• To analyze the performance of the anytime Hedge algorithm in the stochastic setting, where expert losses are i.i.d. and the best expert has a constant gap $\Delta$.
• To determine whether standard Hedge, designed for adversarial settings, can adapt to easier stochastic instances without explicit tuning.
• To compare the anytime Hedge variant with fixed-horizon and doubling-trick variants in terms of adaptivity to stochastic environments.
• To identify the limitations of anytime Hedge under Bernstein conditions, which model low-noise scenarios.
• To evaluate whether more adaptive algorithms provide improvements beyond the gap $\Delta$ in complexity measures.

Proposed method

• The analysis uses a decreasing learning rate $\eta_t = c_0 \sqrt{\log M / t}$, which ensures anytime regret bounds for all $T \geq 1$.
• The regret is decomposed into expected losses over time, leveraging the exponential weighting mechanism of Hedge to bound the influence of suboptimal experts.
• Hoeffding's inequality is applied to control the tail probabilities of cumulative loss differences between the best and suboptimal experts.
• The proof constructs a lower bound instance with constant losses $\ell_{1,t} = 0$, $\ell_{i,t} = \Delta$, satisfying the $(1,1)$-Bernstein condition.
• The analysis derives a lower bound on the expected regret by bounding the probability that the algorithm assigns significant weight to suboptimal experts.
• It compares the regret of Decreasing Hedge to that of more adaptive algorithms under Bernstein conditions, showing suboptimality in certain low-noise regimes.

Experimental results

Research questions

• RQ1 Does the anytime Hedge algorithm with decreasing learning rate adapt to the stochastic setting, achieving better-than-worst-case regret?
• RQ2 How does the performance of anytime Hedge compare to fixed-horizon and doubling-trick variants in the stochastic regime?
• RQ3 Can the standard Hedge algorithm achieve optimal regret under Bernstein conditions, which model low-noise learning?
• RQ4 What are the intrinsic limitations of anytime Hedge in stochastic environments beyond the gap $\Delta$?
• RQ5 Do more adaptive algorithms provide provable improvements over standard Hedge in complex stochastic instances?

Key findings

• The anytime Hedge algorithm with $\eta_t \propto \sqrt{\log M / t}$ achieves an optimal regret bound of $O(\log M / \Delta)$ in the stochastic setting, matching the best adaptive algorithms.
• The fixed-horizon Hedge variant with constant $\eta_t = \sqrt{\log M / T}$ suffers $\Theta(\sqrt{T \log M})$ regret even in the stochastic case, showing it fails to adapt.
• The anytime Hedge algorithm does not achieve improved regret under Bernstein conditions with parameter $\beta < 1$, indicating a fundamental limitation in low-noise regimes.
• A lower bound of $\Omega(1/(c_0^4 (\log M)^2 \Delta))$ is established for the regret of Decreasing Hedge, showing it cannot improve beyond this under certain conditions.
• The analysis confirms that more adaptive algorithms, such as those satisfying second-order bounds, can outperform standard Hedge in complex stochastic instances beyond the gap $\Delta$.