[Paper Review] Online Coordinate Boosting

Raphael Pelossof, Michael Jones+2 more|ArXiv.org|Oct 24, 2008

Face and Expression Recognition10 references4 citations

TL;DR

This paper proposes Online Coordinate Boosting (OCB), a novel online boosting algorithm that approximates batch AdaBoost by incrementally updating weak hypothesis weights using a coordinate-wise minimization of AdaBoost's exponential loss. By deriving an efficient approximation that avoids costly full-dataset computations, OCB achieves better approximation error and generalization performance than Oza and Russell’s algorithm on synthetic, face, and MNIST datasets, with optimal performance at an order of 400.

ABSTRACT

We present a new online boosting algorithm for adapting the weights of a boosted classifier, which yields a closer approximation to Freund and Schapire's AdaBoost algorithm than previous online boosting algorithms. We also contribute a new way of deriving the online algorithm that ties together previous online boosting work. We assume that the weak hypotheses were selected beforehand, and only their weights are updated during online boosting. The update rule is derived by minimizing AdaBoost's loss when viewed in an incremental form. The equations show that optimization is computationally expensive. However, a fast online approximation is possible. We compare approximation error to batch AdaBoost on synthetic datasets and generalization error on face datasets and the MNIST dataset.

Motivation & Objective

• To develop a fast and accurate online learning algorithm that adapts a pre-trained boosted classifier to new data streams and concept drift.
• To address the computational infeasibility of exactly replicating batch AdaBoost in an online setting due to full-dataset re-evaluation requirements.
• To derive a principled online approximation of AdaBoost’s loss minimization by incrementally adding examples and updating weights.
• To improve upon existing online boosting methods like Oza and Russell’s by reducing approximation error while maintaining computational efficiency.

Proposed method

• Derive an exact incremental update rule for AdaBoost’s exponential loss by adding one example at a time to the training set.
• Formulate a fast online approximation that avoids recomputing all weak hypothesis outputs on the full dataset by using greedy coordinate-wise minimization.
• Use a multiplicative weight update rule for weak hypotheses, inspired by Littlestone’s Winnow and Freund and Schapire’s AdaBoost, but adapted for online sequential learning.
• Store and update cumulative sums of correctly and incorrectly classified example weights per weak hypothesis to estimate weighted error rates under memory and speed constraints.
• Introduce a parameter called 'order' that controls the number of past examples used in the approximation, enabling trade-offs between accuracy and computational cost.
• Initialize the algorithm with weights from a small batch of initial data, then continuously adapt to new examples in a streaming fashion.

Experimental results

Research questions

• RQ1 Can an online boosting algorithm be derived that more closely approximates batch AdaBoost than existing methods?
• RQ2 What is the computational cost of maintaining exact online updates to AdaBoost’s weak hypothesis weights, and can it be mitigated with a practical approximation?
• RQ3 How does the proposed online approximation perform in terms of approximation error and generalization error compared to Oza and Russell’s algorithm?
• RQ4 What is the impact of the 'order' parameter on the trade-off between approximation accuracy and computational efficiency?
• RQ5 Can the algorithm be effectively initialized and adapted without distinguishing between offline and online training data?

Key findings

• On synthetic datasets, OCB with order 400 achieved significantly lower approximation error than Oza and Russell’s algorithm, demonstrating better fidelity to batch AdaBoost.
• On the frontal face dataset, OCB with order 400 achieved higher average AUC (area under the ROC curve) than Oza and Russell’s algorithm, indicating superior generalization performance.
• On the MNIST dataset, OCB achieved lower test error (e.g., 0.31% for digit 0 vs. 0.35% for Oza) and lower approximation error (e.g., 0.07% vs. 0.10% for Oza) across all digits.
• The optimal order for both face and MNIST classifiers was found to be 400, balancing approximation accuracy and computational cost.
• The algorithm’s performance was robust to data ordering, as evidenced by consistent results across 10 permuted training sets.
• A trade-off was observed at high orders: while approximation error initially decreased, it deteriorated beyond order 400 due to greedy minimization not optimally reducing total error.