[Paper Review] Online Agnostic Boosting via Regret Minimization

Nataly Brukhim, Xinyi Chen+2 more|arXiv (Cornell University)|Mar 2, 2020

Advanced Bandit Algorithms Research34 references4 citations

TL;DR

This paper presents the first online agnostic boosting algorithm that transforms a weak online learner with sublinear regret into a strong learner with vanishing regret. By reducing the boosting problem to online convex optimization (OCO), the method unifies agnostic and realizable settings in both online and statistical learning, achieving near-optimal correlation bounds via regret minimization.

ABSTRACT

Boosting is a widely used machine learning approach based on the idea of aggregating weak learning rules. While in statistical learning numerous boosting methods exist both in the realizable and agnostic settings, in online learning they exist only in the realizable case. In this work we provide the first agnostic online boosting algorithm; that is, given a weak learner with only marginally-better-than-trivial regret guarantees, our algorithm boosts it to a strong learner with sublinear regret. Our algorithm is based on an abstract (and simple) reduction to online convex optimization, which efficiently converts an arbitrary online convex optimizer to an online booster. Moreover, this reduction extends to the statistical as well as the online realizable settings, thus unifying the 4 cases of statistical/online and agnostic/realizable boosting.

Motivation & Objective

• To close the gap in online learning by developing the first agnostic online boosting algorithm, as prior methods were limited to the realizable (mistake-bound) setting.
• To unify four learning paradigms—statistical/online and agnostic/realizable—through a single reduction framework.
• To enable efficient reductions from differentially private learning to online learning by extending the known realizable-case reduction to the agnostic case.
• To support applications in online control and time series prediction where adversarial sequences are common and strong learners are essential.

Proposed method

• The algorithm reduces online boosting to online convex optimization (OCO), using a convex combination of weak learners' predictions with weights learned via OCO.
• It employs a regret-minimizing OCO algorithm to dynamically adjust the weights of N weak learners based on their past performance.
• The weak learner is assumed to have a marginally better-than-trivial regret guarantee, defined via expected correlation gain over adversarial sequences.
• The final prediction is a normalized average of weak learner outputs, projected to ±1 to ensure valid binary predictions.
• The method applies to both agnostic and realizable settings by adjusting the loss function and label relabeling strategy.
• The framework is general and extends to any OCO algorithm with sublinear regret, enabling modular plug-in of different optimization components.

Experimental results

Research questions

• RQ1 Can a weak online learner with sublinear regret be boosted into a strong learner with vanishing regret in the agnostic online setting?
• RQ2 Is it possible to unify online and statistical learning across agnostic and realizable settings through a single boosting framework?
• RQ3 Can the reduction from differentially private learning to online learning be extended beyond the realizable case using agnostic online boosting?
• RQ4 What is the optimal trade-off between the number of weak learners and the regret bound in online agnostic boosting?
• RQ5 How does the performance of the boosted learner scale with the weak learner's advantage parameter γ and the OCO regret?

Key findings

• The proposed online agnostic boosting algorithm achieves a correlation of at least $1 - O(1/(γ\sqrt{T}))$ with high probability, approaching perfect correlation as T increases.
• The algorithm reduces the problem to online convex optimization, enabling the use of any OCO algorithm with sublinear regret to achieve strong generalization.
• The method unifies all four combinations of learning settings—statistical/online and agnostic/realizable—under a single abstract reduction.
• In the realizable setting, the algorithm achieves a correlation of at least $1 - \tilde{O}(R_{\mathcal{W}}(T)/(\gamma T) + R_{\mathcal{A}}(N)/N)$, which approaches 1 as T and N grow.
• The framework enables efficient reductions from differentially private learning to online learning, extending prior results that were limited to the realizable case.
• The theoretical analysis shows that the regret of the strong learner is sublinear in T, even when the weak learner only has a constant advantage over random guessing.