[Paper Review] Optimal learning with Bernstein Online Aggregation

Olivier Wintenberger|arXiv (Cornell University)|Apr 4, 2014

Advanced Bandit Algorithms Research23 references4 citations

TL;DR

This paper introduces Bernstein Online Aggregation (BOA), an online learning algorithm that achieves optimal fast rate convergence of log(M)/n in deviation for the model selection aggregation problem under square loss. By incorporating a second-order refinement via observable second-order risk terms, BOA overcomes limitations of classical exponential weights and becomes the first online procedure whose batch version attains the optimal fast rate, with a fully adaptive variant achieving near-optimality up to a log log n factor.

ABSTRACT

We introduce a new recursive aggregation procedure called Bernstein Online Aggregation (BOA). The exponential weights include an accuracy term and a second order term that is a proxy of the quadratic variation as in Hazan and Kale (2010). This second term stabilizes the procedure that is optimal in different senses. We first obtain optimal regret bounds in the deterministic context. Then, an adaptive version is the first exponential weights algorithm that exhibits a second order bound with excess losses that appears first in Gaillard et al. (2014). The second order bounds in the deterministic context are extended to a general stochastic context using the cumulative predictive risk. Such conversion provides the main result of the paper, an inequality of a novel type comparing the procedure with any deterministic aggregation procedure for an integrated criteria. Then we obtain an observable estimate of the excess of risk of the BOA procedure. To assert the optimality, we consider finally the iid case for strongly convex and Lipschitz continuous losses and we prove that the optimal rate of aggregation of Tsybakov (2003) is achieved. The batch version of the BOA procedure is then the first adaptive explicit algorithm that satisfies an optimal oracle inequality with high probability.

Motivation & Objective

• To develop an online aggregation procedure that achieves the optimal fast rate of convergence log(M)/n in deviation for the model selection problem.
• To overcome the sub-optimality of classical exponential weights in achieving fast rates by introducing a second-order refinement.
• To design a fully adaptive version of the algorithm that tunes learning rates automatically, improving practicality.
• To establish theoretical guarantees for both the standard and adaptive BOA procedures under the bounded iid setting with Lipschitz and strongly convex losses.

Proposed method

• Proposes Bernstein Online Aggregation (BOA), a recursive aggregation procedure using exponential weights with a second-order refinement term based on cumulative predictive risk estimation.
• Introduces a new stochastic conversion technique to estimate the cumulative predictive risk using observable second-order terms in any stochastic environment.
• Employs a second-order empirical bound derived from the Young inequality and a 'poissonnian' inequality to control the deviation of the predictive risk.
• Uses a temperature-like parameter η to balance the first- and second-order terms in the weight update, with an optimal choice η* that ensures the fast rate.
• Develops a multiple learning rates variant of BOA that adapts to unknown gradient bounds, achieving near-optimality up to a log log n factor.
• Applies the method to the square loss setting with bounded gradients and strongly convex losses, ensuring fast convergence rates.

Experimental results

Research questions

• RQ1 Can an online aggregation procedure achieve the optimal fast rate of log(M)/n in deviation for model selection under the square loss?
• RQ2 Why do classical exponential weights fail to achieve this fast rate, and how can a second-order refinement resolve this?
• RQ3 Is it possible to design an adaptive online algorithm that automatically tunes learning rates without prior knowledge of gradient bounds?
• RQ4 Can the batch version of an online algorithm achieve the optimal fast rate, and if so, under what conditions?
• RQ5 How does the observable second-order term contribute to achieving fast rates in the presence of stochastic and possibly degenerate covariates?

Key findings

• The batch version of BOA achieves the optimal fast rate of convergence log(M)/n in deviation for the model selection problem under the square loss.
• The second-order refinement in BOA is essential for achieving this fast rate, as classical exponential weights cannot attain it.
• The adaptive BOA procedure achieves a rate of convergence within a log log n factor of the optimal rate, making it fully automatic and practical.
• The predictive risk of the online BOA procedure satisfies a high-probability bound of the form min_j R(f_j) + O((log M + x)/n), confirming its optimality.
• The algorithm has O(Mn) computational complexity, making it more efficient than batch procedures requiring iterative optimization.
• The theoretical analysis confirms that the difference between the online and batch risks is negligible compared to the fast rate, supporting the optimality of the online procedure.