[Paper Review] Online Learning with Many Experts

Alon Cohen, Shie Mannor|arXiv (Cornell University)|Feb 25, 2017

Advanced Bandit Algorithms Research27 references3 citations

TL;DR

This paper presents an online learning algorithm for prediction with expert advice that achieves a regret bound of $\widetilde{O}(\epsilon T + \mathcal{N}(\epsilon, L_T) + \sqrt{T\mathcal{N}(\epsilon, L_T)})$, where $\mathcal{N}(\epsilon, L_T)$ is the $\epsilon$-covering number of the sequence of loss functions under the infinity norm. The algorithm adaptively constructs a covering of the loss sequence and automatically tunes $\epsilon$ in hindsight, enabling regret independent of the number of experts, even when it is infinite.

ABSTRACT

We study the problem of prediction with expert advice when the number of experts in question may be extremely large or even infinite. We devise an algorithm that obtains a tight regret bound of $\widetilde{O}(εT + N + \sqrt{NT})$, where $N$ is the empirical $ε$-covering number of the sequence of loss functions generated by the environment. In addition, we present a hedging procedure that allows us to find the optimal $ε$ in hindsight. Finally, we discuss a few interesting applications of our algorithm. We show how our algorithm is applicable in the approximately low rank experts model of Hazan et al. (2016), and discuss the case of experts with bounded variation, in which there is a surprisingly large gap between the regret bounds obtained in the statistical and online settings.

Motivation & Objective

• To address online learning with a very large or infinite number of experts, where traditional regret bounds scale poorly with expert count.
• To develop an algorithm whose regret depends not on the number of experts, but on the empirical $\epsilon$-covering number of the loss sequence.
• To design a hedging procedure that automatically tunes the accuracy parameter $\epsilon$ without prior knowledge.
• To analyze the performance gap between statistical and online settings for experts with bounded variation.
• To demonstrate applicability to structured expert models such as low-rank experts and binary loss settings.

Proposed method

• The algorithm constructs an $\epsilon$-covering of the sequence of loss functions $L_T = (\ell_1, \dots, \ell_T)$ in an online fashion as the losses are revealed.
• It uses a dynamic packing strategy to maintain a set of representative experts that approximate the loss behavior of the full expert set.
• The regret bound is derived using a combination of covering number analysis and concentration inequalities under the infinity norm.
• A hedging procedure is introduced to select the optimal $\epsilon$ in hindsight by balancing $\epsilon T$ and $\sqrt{T \mathcal{N}(\epsilon, L_T)}$ terms.
• The method applies to structured settings such as low-rank experts and bounded-variation experts via appropriate covering number bounds.
• Theoretical guarantees are established using metric entropy and packing arguments in compact metric spaces.

Experimental results

Research questions

• RQ1 Can online learning regret be made independent of the number of experts when the number is large or infinite?
• RQ2 What is the optimal trade-off between approximation error ($\epsilon T$) and complexity ($\mathcal{N}(\epsilon, L_T)$) in online expert learning?
• RQ3 Is it possible to automatically tune the accuracy parameter $\epsilon$ in online learning without prior knowledge?
• RQ4 Why does the regret gap between statistical and online settings become so large for experts with bounded variation?
• RQ5 Can the proposed algorithm achieve tight regret bounds in structured expert models like low-rank or binary loss settings?

Key findings

• The algorithm achieves a regret bound of $\widetilde{O}(\epsilon T + \mathcal{N}(\epsilon, L_T) + \sqrt{T\mathcal{N}(\epsilon, L_T)})$, which is independent of the total number of experts.
• For the low-rank expert model, the covering number $\mathcal{N}(\epsilon, L_T)$ is bounded by $O((2n/\epsilon)^k)$, leading to a regret of $\widetilde{O}(T^{k/2})$ for $k$-sparse experts.
• In the bounded-variation case with $V$ variations, the regret is $\widetilde{O}(T^{V/2})$, which is trivial for $V \geq 2$, showing a fundamental gap between statistical and online settings.
• A lower bound shows that any learner must incur at least $T$ expected regret when $T < \log K$ and $V \leq 2$, even for constant variation.
• The hedging procedure enables automatic selection of $\epsilon$ without prior knowledge, achieving optimal trade-off in hindsight.
• The paper identifies a key open problem: finding algorithms with tight regret bounds in both high-dimensional and low-complexity environments.