[Paper Review] Credit card score prediction using machine learning models: A new dataset

Anas Arram, Masri Ayob+3 more|arXiv (Cornell University)|Oct 4, 2023

Financial Distress and Bankruptcy Prediction4 citations

TL;DR

This study introduces a new credit card scoring dataset from an American bank and evaluates multiple machine learning models—logistic regression, XGBoost, LightGBM, decision trees, random forests, and MLP—for default prediction. The MLP model achieved the highest recall (80%) and AUC (86.7%), making it the best-performing model for minimizing false negatives in credit risk assessment.

ABSTRACT

The use of credit cards has recently increased, creating an essential need for credit card assessment methods to minimize potential risks. This study investigates the utilization of machine learning (ML) models for credit card default prediction system. The main goal here is to investigate the best-performing ML model for new proposed credit card scoring dataset. This new dataset includes credit card transaction histories and customer profiles, is proposed and tested using a variety of machine learning algorithms, including logistic regression, decision trees, random forests, multi-layer perceptron (MLP) neural network, XGBoost, and LightGBM. To prepare the data for machine learning models, we perform data pre-processing, feature extraction, feature selection, and data balancing techniques. Experimental results demonstrate that MLP outperforms logistic regression, decision trees, random forests, LightGBM, and XGBoost in terms of predictive performance in true positive rate, achieving an impressive area under the curve (AUC) of 86.7% and an accuracy rate of 91.6%, with a recall rate exceeding 80%. These results indicate the superiority of MLP in predicting the default customers and assessing the potential risks. Furthermore, they help banks and other financial institutions in predicting loan defaults at an earlier stage.

Motivation & Objective

• To develop a new, comprehensive credit card scoring dataset incorporating transaction histories and customer profiles for improved default prediction.
• To evaluate the performance of diverse machine learning models in predicting credit card defaults with high recall to minimize false negatives.
• To address data quality issues such as missing values, outliers, and class imbalance through systematic preprocessing techniques.
• To identify the optimal machine learning model for early detection of defaulting customers, prioritizing recall over other metrics.
• To provide financial institutions with a robust, data-driven tool for assessing credit risk and reducing exposure to loan defaults.

Proposed method

• Collected and curated a new credit card scoring dataset from an American bank, integrating transaction histories and customer demographic information.
• Applied data preprocessing techniques including feature extraction, imputation for missing values, outlier treatment using IQR, and SMOTE for class balancing.
• Trained and evaluated six machine learning models: logistic regression, decision trees, random forests, MLP neural networks, XGBoost, and LightGBM.
• Used standard evaluation metrics including accuracy, AUC, precision, and recall, with a focus on recall as the primary performance criterion.
• Visualized model performance using ROC curves to compare discrimination ability across models.
• Selected the best-performing model based on recall and AUC, with additional analysis of precision-recall trade-offs.

Figure 1: Feed forward neural network structure [ 32 ]

Experimental results

Research questions

• RQ1 Which machine learning model achieves the highest recall in predicting credit card defaults on the newly proposed dataset?
• RQ2 How do different preprocessing techniques—such as SMOTE and outlier handling—affect the performance of credit scoring models?
• RQ3 What is the comparative performance of traditional models (e.g., logistic regression) versus advanced models (e.g., MLP, LightGBM) in credit default prediction?
• RQ4 Can deep learning models like MLP outperform tree-based models in identifying true default cases, especially when recall is prioritized?
• RQ5 How does model performance vary across key metrics (AUC, accuracy, precision, recall), and which model offers the best balance for real-world credit risk assessment?

Key findings

• The MLP classifier achieved the highest recall (80%) among all models, making it the most effective at identifying true defaulting customers.
• The MLP model recorded an AUC of 86.7%, indicating strong discrimination between default and non-default cases.
• LightGBM achieved the highest accuracy (94.44%) and AUC (84.19%), but had a lower recall (70%) compared to MLP.
• Random Forest achieved high accuracy (93.06%) but suffered from low recall (75%) and precision (75%), indicating a trade-off in identifying all positive cases.
• Logistic regression and decision tree classifiers underperformed, with recall values of 70% and 70%, respectively, and lower AUC scores.
• The ROC curve analysis confirmed that MLP and Random Forest had curves closest to the upper-left corner, indicating superior overall performance in distinguishing defaulters.

Figure 2: The distribution of class labels