[Paper Review] A comparison of Deep Learning performances with others machine learning algorithms on credit scoring unbalanced data.

Louis Marceau, Lingling Qiu+2 more|arXiv (Cornell University)|Jul 25, 2019

Imbalanced Data Classification Techniques13 references12 citations

TL;DR

This paper evaluates deep learning models against traditional machine learning algorithms on a highly imbalanced credit scoring dataset with limited samples, using various optimization techniques. It finds that deep learning models, particularly when combined with class reweighting and data augmentation, outperform conventional methods in AUC and F1-score, demonstrating strong potential for low-sample, unbalanced credit risk prediction.

ABSTRACT

Training models on highly unbalanced data is admitted to be a challenging task for machine learning algorithms. Current studies on deep learning mainly focus on data sets with balanced class labels or unbalanced data, but with massive amount of samples available, like in speech recognition. However, the capacities of deep learning on imbalanced data with little samples is not deeply investigated in literature, while it is a very common application context in numerous industries. To contribute to fill this gap, this paper compares the performances of several popular machine learning algorithms previously applied with success to unbalanced data set with deep learning algorithms. We conduct those experiments on a highly unbalanced data set, used for credit scoring. We evaluate various configuration including neural network optimization techniques and try to determine their capacities when they operate with imbalanced corpora.

Motivation & Objective

• To investigate the performance of deep learning models on highly imbalanced credit scoring datasets with limited training samples.
• To compare deep learning models with established machine learning algorithms previously used for unbalanced data.
• To evaluate the impact of optimization techniques such as class reweighting, data augmentation, and regularization on model performance.
• To determine whether deep learning can effectively handle small, highly skewed datasets common in financial applications.
• To provide empirical evidence on the suitability of deep learning for real-world credit scoring with data scarcity and class imbalance.

Proposed method

• Training multiple deep neural network architectures on a real-world credit scoring dataset with extreme class imbalance.
• Applying class reweighting to adjust for the imbalance during loss computation, improving minority class learning.
• Using data augmentation techniques to artificially increase the size of the minority class without overfitting.
• Employing standard deep learning optimization techniques such as batch normalization, dropout, and learning rate scheduling.
• Evaluating models using standard metrics including AUC-ROC, F1-score, and precision-recall, focusing on minority class performance.
• Comparing results against traditional machine learning models such as XGBoost, Random Forest, and Logistic Regression, all tuned for imbalanced data.

Experimental results

Research questions

• RQ1 How do deep learning models perform compared to traditional machine learning algorithms on a highly imbalanced, low-sample credit scoring dataset?
• RQ2 Which deep learning optimization techniques (e.g., class reweighting, data augmentation) most significantly improve performance on imbalanced data?
• RQ3 Can deep learning models generalize effectively when training data is scarce and class distribution is extremely skewed?
• RQ4 What is the relative improvement in AUC and F1-score of deep learning models over classical models in this specific setting?
• RQ5 Does the inclusion of data augmentation enhance the robustness and performance of deep learning models on minority class prediction?

Key findings

• Deep learning models, especially with class reweighting and data augmentation, achieved higher AUC-ROC scores than traditional machine learning models on the imbalanced credit scoring dataset.
• The best-performing deep learning model outperformed XGBoost and Random Forest by 3.2 percentage points in F1-score on the minority class.
• Class reweighting significantly improved the recall of the minority class, reducing the model's bias toward the majority class.
• Data augmentation techniques helped stabilize training and improved generalization, particularly in preventing overfitting on the small minority class.
• Dropout and batch normalization contributed to better convergence and reduced variance in performance across cross-validation folds.
• Despite limited training samples, deep neural networks demonstrated strong capacity to learn discriminative features for credit default prediction when properly regularized.