[Paper Review] Optimal Weak to Strong Learning

Kasper Green Larsen, Martin Ritzert|arXiv (Cornell University)|Jun 3, 2022

Machine Learning and Algorithms4 citations

TL;DR

This paper presents a new weak-to-strong learning algorithm that achieves optimal sample complexity by eliminating two logarithmic factors present in AdaBoost, reducing the required training data to the theoretical minimum. The method uses sub-sampling and margin-based voting, with a matching lower bound proving its optimality for VC-dimension $d$ and advantage $\gamma$ in the regime $2^{-d} < \gamma < 1/80$. The result closes the long-standing question of the exact sample complexity for weak-to-strong learning.

ABSTRACT

The classic algorithm AdaBoost allows to convert a weak learner, that is an algorithm that produces a hypothesis which is slightly better than chance, into a strong learner, achieving arbitrarily high accuracy when given enough training data. We present a new algorithm that constructs a strong learner from a weak learner but uses less training data than AdaBoost and all other weak to strong learners to achieve the same generalization bounds. A sample complexity lower bound shows that our new algorithm uses the minimum possible amount of training data and is thus optimal. Hence, this work settles the sample complexity of the classic problem of constructing a strong learner from a weak learner.

Motivation & Objective

• To resolve the long-standing open problem of determining the exact sample complexity for converting a weak learner into a strong learner.
• To design a learning algorithm that achieves the minimal possible number of training samples required to reach a given generalization error and confidence level.
• To establish a tight lower bound on sample complexity that matches the upper bound of the proposed algorithm, proving optimality.
• To improve upon AdaBoost's sample complexity by eliminating two logarithmic factors in the dependence on $\varepsilon$, $\gamma$, and $d$.
• To analyze the generalization performance of voting classifiers with large margins using a novel information-theoretic approach.

Proposed method

• The algorithm constructs a strong learner by generating $m^{0.79}$ sub-samples of linear size from the training data, each used to train a weak learner.
• It applies a modified version of AdaBoost$_\nu^*$ on each sub-sample to produce a set of hypotheses with large margins.
• The final hypothesis is formed by taking a majority vote over all sub-sample predictions, leveraging the margin-based generalization bound.
• The analysis uses the Littlewood-Offord lemma to bound the entropy of the concept class conditioned on hypothesis performance, enabling tighter generalization bounds.
• A novel information-theoretic argument combines entropy and conditional entropy to derive a lower bound on the probability of poor generalization, leading to the sample complexity lower bound.
• The method proves that any weak-to-strong learning algorithm must use $\Omega\left(\frac{d}{\varepsilon\gamma^2} + \frac{\ln(1/\delta)}{\varepsilon}\right)$ samples, matching the upper bound of the proposed algorithm.

Experimental results

Research questions

• RQ1 What is the exact sample complexity required to convert a $\gamma$-weak learner into a strong learner with error $\varepsilon$ and confidence $1-\delta$?
• RQ2 Can the sample complexity of AdaBoost be improved by removing logarithmic factors in $d$, $\varepsilon$, and $\gamma$?
• RQ3 Is there a matching lower bound that proves the optimality of the proposed algorithm’s sample complexity?
• RQ4 Can the sub-sampling strategy be optimized to reduce the number of calls to the weak learner for large datasets?
• RQ5 Is a majority of majorities necessary for optimal generalization, or can a simpler voting rule achieve the same bound?

Key findings

• The proposed algorithm achieves a sample complexity of $O\left(\frac{d}{\varepsilon\gamma^2} + \frac{\ln(1/\delta)}{\varepsilon}\right)$, improving upon AdaBoost’s bound by removing two logarithmic factors.
• A matching lower bound of $\Omega\left(\frac{d}{\varepsilon\gamma^2} + \frac{\ln(1/\delta)}{\varepsilon}\right)$ is proven, showing that the algorithm is optimal up to constant factors.
• The analysis establishes a new generalization bound for voting classifiers with large margins, which is tighter than existing $k$-th margin bounds for constant error.
• The use of the Littlewood-Offord lemma in the information-theoretic argument enables a sharp bound on the probability of poor generalization, crucial for the lower bound proof.
• The result settles the sample complexity of weak-to-strong learning in the realizable case, up to constant factors, for $\gamma > 2^{-d}$.
• The paper identifies that for $\gamma < 2^{-d}$, the current lower bound does not apply, suggesting a potentially better dependency on $\gamma$ might exist, as hinted by recent results on VC-dimension growth.