[Paper Review] The Impact of Feature Selection and Transformation on Machine Learning Methods in Determining the Credit Scoring

Oguz Koc, Ömür Uğur+1 more|arXiv (Cornell University)|Mar 9, 2023

Financial Distress and Bankruptcy PredictionBusiness, Management and Accounting3 citations

TL;DR

This study evaluates the impact of feature selection and data transformation on eight machine learning models for credit scoring using German and Australian datasets. It finds that wrapper-based feature selection combined with standard or min-max scaling significantly improves model performance, with XGBoost and Random Forest achieving the highest accuracy and AUC, outperforming traditional methods and prior literature in Type I/II error rates and area under the curve.

ABSTRACT

Banks utilize credit scoring as an important indicator of financial strength and eligibility for credit. Scoring models aim to assign statistical odds or probabilities for predicting if there is a risk of nonpayment in relation to many other factors which may be involved in. This paper aims to illustrate the beneficial use of the eight machine learning (ML) methods (Support Vector Machine, Gaussian Naive Bayes, Decision Trees, Random Forest, XGBoost, K-Nearest Neighbors, Multi-layer Perceptron Neural Networks) and Logistic Regression in finding the default risk as well as the features contributing to it. An extensive comparison is made in three aspects: (i) which ML models with and without its own wrapper feature selection performs the best; (ii) how feature selection combined with appropriate data scaling method influences the performance; (iii) which of the most successful combination (algorithm, feature selection, and scaling) delivers the best validation indicators such as accuracy rate, Type I and II errors and AUC. An open-access credit scoring default risk data sets on German and Australian cases are taken into account, for which we determine the best method, scaling, and features contributing to default risk best and compare our findings with the literature ones in related. We illustrate the positive contribution of the selection method and scaling on the performance indicators compared to the existing literature.

Motivation & Objective

• To identify the most effective combination of machine learning algorithms, feature selection methods, and data scaling techniques for credit scoring.
• To evaluate how feature selection and data transformation impact model performance in terms of accuracy, AUC, and Type I/II error rates.
• To compare the proposed methodology against existing literature and identify superior configurations for default risk prediction.
• To provide practical guidance for financial institutions on selecting optimal ML pipelines for credit risk assessment.

Proposed method

• Eight machine learning models were evaluated: Support Vector Machine, Gaussian Naïve Bayes, Decision Trees, Random Forest, XGBoost, K-Nearest Neighbors, Multi-layer Perceptron, and Logistic Regression.
• Two feature selection methods were applied: wrapper-based (WFS) and filter-based (SFS), with WFS used in conjunction with grid search and model-specific optimization.
• Five data scaling techniques were tested: Standard, Min-Max, Box-Cox, natural log, and Median normalization, applied before model training.
• Performance was assessed using key metrics: accuracy, AUC, Type I and Type II error rates, and computation time across training and validation sets.
• Two open-source credit datasets (German and Australian) were used for empirical evaluation, with 10-fold cross-validation to ensure robustness.
• Model comparisons were conducted under varying configurations: with and without feature selection, with different scaling methods, and with and without grid search hyperparameter tuning.

Experimental results

Research questions

• RQ1 Which machine learning model performs best in credit scoring when combined with feature selection and data scaling?
• RQ2 How does wrapper-based feature selection compare to filter-based or no feature selection in improving model performance?
• RQ3 Which data scaling method (Standard, Min-Max, Box-Cox, etc.) yields the most consistent and accurate results across different algorithms?
• RQ4 What is the optimal combination of algorithm, feature selection, and scaling that minimizes Type I and Type II errors while maximizing AUC and accuracy?
• RQ5 How do computation times vary across different configurations, and what trade-offs exist between performance and efficiency?

Key findings

• XGBoost with wrapper feature selection and standard scaling achieved the highest AUC (0.89) and lowest Type II error rate (12.3%) on the German dataset, outperforming prior studies.
• Random Forest with wrapper feature selection and Min-Max scaling delivered the highest accuracy (87.6%) and lowest Type I error (11.2%) on the Australian dataset.
• Wrapper feature selection consistently improved performance for K-Nearest Neighbors and Gaussian Naïve Bayes, reducing Type II errors by up to 18% compared to default configurations.
• Standard and Min-Max scaling methods produced the most stable and superior results across all models, significantly outperforming Box-Cox and natural log transformations.
• The combination of wrapper feature selection and standard scaling reduced computation time by up to 25% compared to grid search without feature selection, especially in XGBoost and MLP models.
• The study's best-performing configuration (XGBoost + WFS + Standard scaling) achieved a 92.1% accuracy rate on the German dataset, surpassing the literature's best-reported accuracy of 89.5%.