[Paper Review] A Bayesian Boosting Model

Alexander Lorbert, David M. Blei+2 more|arXiv (Cornell University)|Sep 10, 2012

Machine Learning and Data Classification22 references3 citations

TL;DR

This paper proposes VIBoost, a Bayesian boosting algorithm that models label noise hierarchically using variational inference to improve robustness against noisy data. By embedding AdaBoost in a probabilistic framework with latent noise variables, VIBoost achieves performance comparable to AdaBoost while providing interpretable noise statistics and a smoothed update rule closely related to AdaBoost.

ABSTRACT

We offer a novel view of AdaBoost in a statistical setting. We propose a Bayesian model for binary classification in which label noise is modeled hierarchically. Using variational inference to optimize a dynamic evidence lower bound, we derive a new boosting-like algorithm called VIBoost. We show its close connections to AdaBoost and give experimental results from four datasets.

Motivation & Objective

• To address the vulnerability of AdaBoost to label noise in real-world datasets where mislabeled examples are common.
• To develop a principled Bayesian framework that models label noise as a hierarchical latent variable, enabling uncertainty quantification.
• To derive a new boosting-like algorithm through variational inference that balances classification accuracy and noise robustness.
• To show that the proposed algorithm reduces to a smoothed version of AdaBoost under certain conditions, linking Bayesian inference to classical boosting.
• To empirically validate the method on four datasets, demonstrating competitive performance and meaningful noise characterization.

Proposed method

• Formulates binary classification as a Bayesian model with latent weights on base classifiers and hierarchical noise modeling for label perturbations.
• Uses variational inference to approximate the posterior distribution over classifier weights and noise parameters, optimizing a dynamic evidence lower bound (ELBO).
• Applies a warm-start strategy in iterative inference, introducing one base classifier per iteration to build a dynamic model, similar to AdaBoost's greedy selection.
• Introduces a smoothed update rule derived from the variational objective, which reduces to a regularized version of AdaBoost when noise is negligible.
• Employs numerical integration and mode approximation techniques to compute the log-normalizing constant in the variational approximation.
• Uses a generative process where labels are conditionally distributed via a logistic function of the weighted sum of base classifiers, with noise modeled as random label inversions.

Experimental results

Research questions

• RQ1 Can a fully Bayesian formulation of boosting improve robustness to label noise compared to classical AdaBoost?
• RQ2 How does the inclusion of latent noise variables in a hierarchical model affect the performance and generalization of a boosting algorithm?
• RQ3 What is the relationship between the proposed VIBoost algorithm and AdaBoost, particularly in terms of update rules and convergence behavior?
• RQ4 Can variational inference in this model yield interpretable noise statistics that reflect data quality and classifier reliability?
• RQ5 Does the proposed method maintain competitive classification accuracy while providing uncertainty estimates under noisy conditions?

Key findings

• VIBoost achieves classification performance on par with AdaBoost across four benchmark datasets, demonstrating robustness without sacrificing accuracy.
• The algorithm produces interpretable noise statistics that quantify the level of label perturbation, offering insights into data quality and model reliability.
• In the limit of negligible noise, the VIBoost update rule reduces to a smoothed version of AdaBoost, establishing a theoretical link between Bayesian inference and classical boosting.
• The use of variational inference enables efficient approximate posterior estimation over classifier weights and noise parameters, making the method scalable to moderate-sized datasets.
• Experiments show that VIBoost is less sensitive to noisy examples than AdaBoost, particularly when label noise is significant, due to the explicit modeling of uncertainty.
• The method successfully captures the effect of confusion in real-world data, such as the butterfly ballot scenario, by modeling label noise as random inversions.