[Paper Review] Gradient Boosting Decision Trees on Medical Diagnosis over Tabular Data

Aytaç Yıldız, Arzu Kalaycı|arXiv (Cornell University)|Sep 25, 2024

Artificial Intelligence in HealthcareHealth Professions6 citations

TL;DR

The paper empirically shows that Gradient Boosting Decision Trees (LightGBM, XGBoost, CatBoost) outperform traditional ML and tabular DL models across seven medical tabular datasets, with favorable training times.

ABSTRACT

Medical diagnosis is a crucial task in the medical field, in terms of providing accurate classification and respective treatments. Having near-precise decisions based on correct diagnosis can affect a patient's life itself, and may extremely result in a catastrophe if not classified correctly. Several traditional machine learning (ML), such as support vector machines (SVMs) and logistic regression, and state-of-the-art tabular deep learning (DL) methods, including TabNet and TabTransformer, have been proposed and used over tabular medical datasets. Additionally, due to the superior performances, lower computational costs, and easier optimization over different tasks, ensemble methods have been used in the field more recently. They offer a powerful alternative in terms of providing successful medical decision-making processes in several diagnosis tasks. In this study, we investigated the benefits of ensemble methods, especially the Gradient Boosting Decision Tree (GBDT) algorithms in medical classification tasks over tabular data, focusing on XGBoost, CatBoost, and LightGBM. The experiments demonstrate that GBDT methods outperform traditional ML and deep neural network architectures and have the highest average rank over several benchmark tabular medical diagnosis datasets. Furthermore, they require much less computational power compared to DL models, creating the optimal methodology in terms of high performance and lower complexity.

Motivation & Objective

• Evaluate the performance of GBDT models (XGBoost, LightGBM, CatBoost) on diverse tabular medical diagnosis datasets.
• Compare GBDTs against traditional ML and state-of-the-art tabular DL models.
• Analyze training time and performance trade-offs for practical clinical deployment.
• Provide guidance on model selection for medical tabular data based on dataset size and characteristics.

Proposed method

• Preprocess data with ordinal encoding for categoricals and standardize numerical features.
• Evaluate 5 traditional ML models, 5 DL models, and 4 ensemble models (3 GBDTs) using ROC AUC as metric.
• Perform 8-fold stratified cross-validation to assess generalization.
• Hyperparameter optimization: evaluate ~36 combinations per model based on average ROC AUC across folds.
• Compare models in terms of performance and average training time.

Experimental results

Research questions

• RQ1 Do GBDT models achieve higher ROC AUC than traditional ML and tabular DL models across diverse medical datasets?
• RQ2 Which GBDT implementation (XGBoost, LightGBM, CatBoost) offers the best trade-off between performance and training time?
• RQ3 How do model performances scale with dataset size and feature dimensionality in medical tabular data?
• RQ4 What are the practical implications for model selection in clinical decision support based on accuracy and efficiency?

Key findings

• GBDT models consistently outperform traditional ML and state-of-the-art tabular DL models across all seven datasets.
• LightGBM often achieves the best average ROC AUC and favorable training times among the evaluated models.
• On average, GBDTs provide superior performance with lower computational cost compared to DL architectures.
• Among models, the top-performing GBDT variant varies by dataset, but LightGBM frequently ranks highly and shows strong overall metrics.
• DL models tend to have longer training times due to model complexity, while GBDTs strike a balance between accuracy and efficiency.