[Paper Review] A more robust boosting algorithm

Yoav Freund|ArXiv.org|May 13, 2009

Industrial Vision Systems and Defect Detection14 references91 citations

TL;DR

This paper introduces Robustboost, a novel boosting algorithm designed to be significantly more resilient to label noise than Adaboost and Logitboost. By using a non-convex, dynamically changing potential function that down-weights examples with large negative margins, Robustboost avoids overfitting to noisy labels and achieves superior generalization, especially in high-noise settings.

ABSTRACT

We present a new boosting algorithm, motivated by the large margins theory for boosting. We give experimental evidence that the new algorithm is significantly more robust against label noise than existing boosting algorithm.

Motivation & Objective

• Address the well-known sensitivity of Adaboost and Logitboost to random label noise, which degrades performance rapidly.
• Overcome the theoretical limitation of convex potential functions in boosting, which Long and Servedio showed can be defeated by adversarial noise.
• Develop a new boosting algorithm that maintains high accuracy even when a significant fraction of training labels are corrupted.
• Improve generalization by focusing on examples near the decision boundary and down-weighting those with large negative margins.
• Demonstrate empirically that Robustboost achieves better test error and margin reliability than existing methods under label noise.

Proposed method

• Design a potential-based boosting framework using a non-convex, time-varying potential function that adapts during training.
• Base the algorithm on Freund’s Boost-by-Majority and Brownboost, integrating principles from gradient descent on non-convex potentials.
• Introduce a threshold parameter θ and a noise tolerance parameter ε to control the influence of examples with large negative margins.
• Modify the weight update rule to limit the impact of mislabeled examples with large negative margins, preventing them from dominating the learning process.
• Use a score function s(x) = α·h(x) and define the margin as m(x,y) = y·s(x), with the algorithm focusing on minimizing errors on examples with small to moderate margins.
• Terminate early when convergence is reached, typically within 100–300 iterations under high noise, due to the algorithm’s robustness to label corruption.

Experimental results

Research questions

• RQ1 Can a boosting algorithm be designed to be more robust than Adaboost and Logitboost against random label noise?
• RQ2 Does using a non-convex, adaptive potential function improve generalization in the presence of label noise compared to convex potential functions?
• RQ3 Can examples with large negative margins be effectively down-weighted without sacrificing performance on correctly labeled data?
• RQ4 How does Robustboost perform relative to Logitboost and Adaboost on synthetic datasets with controlled label noise?
• RQ5 To what extent can Robustboost maintain high-confidence predictions on examples with large margins even when labels are corrupted?

Key findings

• On the Long/Servedio synthetic dataset with 10% label noise, Robustboost achieved a test error of 13.5±0.8 with decision stumps, outperforming Logitboost (15.9±0.9) and Adaboost (19.3±1.0).
• With 20% label noise, Robustboost reduced test error to 23.8±1.1 (stumps), compared to 29.4±1.2 for Adaboost and 26.7±1.3 for Logitboost.
• On the Mease/Wyner dataset with 10% noise, Robustboost achieved a clean error rate of 4.3±0.4 on high-margin examples, significantly lower than Logitboost (7.1±0.7) and Adaboost (11.5±1.1).
• Robustboost correctly identified and down-weighted most mislabeled examples with large negative margins, especially in high-noise regimes.
• The algorithm terminated early (100–300 iterations) under high noise, indicating faster convergence and reduced overfitting compared to standard boosting.
• The fraction of low-margin examples (|score| < θ) remained stable at around 10%, while error on high-margin examples was consistently lower than baseline methods, indicating reliable predictions on confident examples.