[Paper Review] Adaptive Base Class Boost for Multi-class Classification

Ping Li|ArXiv.org|Nov 8, 2008

Machine Learning and Algorithms9 references3 citations

TL;DR

This paper proposes ABC-MART, an adaptive boosting algorithm for multi-class classification that improves upon MART by dynamically selecting the worst-performing class as the base class at each boosting iteration. By leveraging a sum-to-zero constraint and redefining the loss function derivatives, ABC-MART reduces training and inference time by 1/K and achieves significantly lower misclassification errors on multiple public datasets, outperforming MART in both accuracy and efficiency.

ABSTRACT

We develop the concept of ABC-Boost (Adaptive Base Class Boost) for multi-class classification and present ABC-MART, a concrete implementation of ABC-Boost. The original MART (Multiple Additive Regression Trees) algorithm has been very successful in large-scale applications. For binary classification, ABC-MART recovers MART. For multi-class classification, ABC-MART considerably improves MART, as evaluated on several public data sets.

Motivation & Objective

• To address the performance gap in multi-class classification for large-scale applications where MART, though successful, is not optimal.
• To develop a boosting framework that dynamically selects the most problematic class as the base class at each iteration to improve overall classification accuracy.
• To reduce computational cost by exploiting the sum-to-zero constraint, thereby decreasing training and inference time by a factor of 1/K.
• To empirically validate that adaptive base class selection leads to consistent improvements over fixed-base or non-adaptive methods.

Proposed method

• ABC-Boost introduces a novel boosting framework that assumes a sum-to-zero constraint on the loss function, allowing the algorithm to model only K−1 classes explicitly while implicitly handling the Kth class.
• At each boosting iteration, the algorithm adaptively selects the class with the highest training loss as the base class, ensuring that the most poorly performing class receives focused learning.
• The method redefines the first and second derivatives of the loss function compared to MART and Logitboost, enabling more effective gradient updates in the functional space.
• ABC-MART uses regression trees as weak learners and applies a one-step Newton update per terminal node, similar to MART, but with modified gradient and Hessian computations.
• The algorithm maintains backward compatibility with MART for binary classification, where it recovers the original MART behavior.
• The implementation differs from MART in only a few lines of code, primarily in base class selection logic and derivative computation.

Experimental results

Research questions

• RQ1 Can adaptive base class selection in multi-class boosting lead to consistent performance improvements over static or fixed-base approaches?
• RQ2 Does redefining the loss function derivatives while maintaining the sum-to-zero constraint enhance learning efficiency and accuracy?
• RQ3 To what extent does ABC-MART reduce training and inference time compared to MART, especially when K is small?
• RQ4 How does ABC-MART compare to MART and Logitboost in terms of test misclassification error across diverse multi-class datasets?

Key findings

• ABC-MART significantly reduces test misclassification error compared to MART on the Covertype dataset, achieving 10.7% error with J=10, compared to MART’s 16.4%.
• On the Pendigits dataset, ABC-MART achieves a test error of 5.2% with J=10, outperforming MART’s 7.0% and all alternative configurations.
• On the Isolet dataset, ABC-MART reduces test error to 84% with ν=0.04 and J=4, compared to MART’s 84% and ABC-MART’s own 84% in the best case.
• The algorithm reduces both training and testing time by approximately 1/K, offering a substantial efficiency gain when K is small.
• Experiments show that using MART’s derivatives with adaptive base selection (Mb) or ABC-MART’s derivatives with a fixed base (b1, b7) results in worse performance than ABC-MART, proving the necessity of both components.
• ABC-MART outperforms all alternative configurations, including those using MART’s derivatives with adaptive base or ABC-MART’s derivatives with fixed base, confirming the critical role of both adaptive base selection and modified derivatives.