[Paper Review] Robust-GBDT: GBDT with Nonconvex Loss for Tabular Classification in the Presence of Label Noise and Class Imbalance

Jiaqi Luo, Yuedong Quan+1 more|arXiv (Cornell University)|Oct 8, 2023

Image and Signal Denoising MethodsComputer Science3 citations

TL;DR

This paper proposes Robust-GBDT, a novel gradient boosting model that integrates nonconvex robust loss functions—particularly Robust Focal Loss—into advanced GBDT frameworks to enhance robustness against label noise and class imbalance in tabular classification. By proving that nonconvex losses can be effectively used in second-order GBDT via Hessian constraints, the method achieves superior generalization and computational efficiency, outperforming existing noise-robust and imbalanced learning techniques.

ABSTRACT

Dealing with label noise in tabular classification tasks poses a persistent challenge in machine learning. While robust boosting methods have shown promise in binary classification, their effectiveness in complex, multi-class scenarios is often limited. Additionally, issues like imbalanced datasets, missing values, and computational inefficiencies further complicate their practical utility. This study introduces Robust-GBDT, a groundbreaking approach that combines the power of Gradient Boosted Decision Trees (GBDT) with the resilience of nonconvex loss functions against label noise. By leveraging local convexity within specific regions, Robust-GBDT demonstrates unprecedented robustness, challenging conventional wisdom. Through seamless integration of advanced GBDT with a novel Robust Focal Loss tailored for class imbalance, Robust-GBDT significantly enhances generalization capabilities, particularly in noisy and imbalanced datasets. Notably, its user-friendly design facilitates integration with existing open-source code, enhancing computational efficiency and scalability. Extensive experiments validate Robust-GBDT's superiority over other noise-robust methods, establishing a new standard for accurate classification amidst label noise. This research heralds a paradigm shift in machine learning, paving the way for a new era of robust and precise classification across diverse real-world applications.

Motivation & Objective

• To address the limitations of existing robust boosting methods that are mostly restricted to binary classification and struggle with class imbalance and computational efficiency.
• To extend the applicability of robust loss functions to multi-class tabular classification by enabling their use in second-order GBDT models.
• To improve generalization under label noise and class imbalance through a novel theoretical insight on Hessian-based optimization with nonconvex losses.
• To develop a user-friendly, efficient, and integrable model that seamlessly replaces standard objective functions in existing GBDT libraries.

Proposed method

• Theoretical analysis shows that the Hessian of the loss function in second-order GBDT need not be globally convex, only convex within a local region, enabling the use of nonconvex robust losses.
• Robust-GBDT integrates advanced GBDT frameworks (e.g., XGBoost, LightGBM) with robust loss functions, leveraging Newton’s method with modified Hessian constraints.
• A new robust loss function, Robust Focal Loss (RFL), is proposed to explicitly address class imbalance by down-weighting easy, majority-class samples.
• The method allows plug-and-play integration into existing open-source GBDT codebases by replacing the objective function with RFL, preserving training speed and scalability.
• The model uses a pair-flipping matrix to simulate label noise in multi-class datasets, enabling controlled evaluation under realistic noise conditions.
• Empirical validation uses standard benchmark datasets with controlled noise rates and imbalanced ratios to evaluate robustness and generalization.

Experimental results

Research questions

• RQ1 Can nonconvex loss functions be effectively used in second-order GBDT models without compromising optimization stability?
• RQ2 How does integrating robust loss functions into GBDT improve performance under label noise and class imbalance in multi-class tabular data?
• RQ3 To what extent does Robust-GBDT outperform existing noise-robust and imbalanced learning methods in terms of accuracy and generalization?
• RQ4 Can the proposed method be efficiently integrated into existing GBDT libraries without sacrificing training speed or requiring architectural changes?

Key findings

• Robust-GBDT achieves significantly higher classification accuracy than baseline GBDT and other noise-robust methods under varying label noise rates across multiple datasets.
• The model demonstrates superior generalization on both binary and multi-class tabular datasets, especially under high noise and strong class imbalance.
• Robust Focal Loss effectively mitigates the negative impact of class imbalance, improving performance on minority classes without degrading overall accuracy.
• Theoretical analysis confirms that nonconvex losses can be safely used in GBDT if the Hessian remains positive definite in the relevant region, enabling broader loss function design.
• Robust-GBDT maintains the computational efficiency of standard GBDT models and can be easily deployed via simple objective function replacement in existing codebases.
• Extensive experiments on 18 benchmark datasets confirm that Robust-GBDT consistently outperforms state-of-the-art baselines in terms of robustness and predictive performance.