[Paper Review] Advanced Payment Security System:XGBoost, LightGBM and SMOTE Integrated

Qi Zheng, Chang Yu+4 more|arXiv (Cornell University)|Jun 7, 2024

IPv6, Mobility, Handover, Networks, Security16 citations

TL;DR

The paper evaluates XGBoost, LightGBM, and CatBoost with SMOTE for fraud detection in payments, showing XGBoost+SMOTE as the top performer across key metrics.

ABSTRACT

With the rise of various online and mobile payment systems, transaction fraud has become a significant threat to financial security. This study explores the application of advanced machine learning models, specifically based on XGBoost and LightGBM, for developing a more accurate and robust Payment Security Protection Model. To enhance data reliability, we meticulously processed the data sources and applied SMOTE (Synthetic Minority Over-sampling Technique) to address class imbalance and improve data representation. By selecting highly correlated features, we aimed to strengthen the training process and boost model performance. We conducted thorough performance evaluations of our proposed models, comparing them against traditional methods including Random Forest, Neural Network, and Logistic Regression. Using metrics such as Precision, Recall, and F1 Score, we rigorously assessed their effectiveness. Our detailed analyses and comparisons reveal that the combination of SMOTE with XGBoost and LightGBM offers a highly efficient and powerful mechanism for payment security protection. Moreover, the integration of XGBoost and LightGBM in a Local Ensemble model further demonstrated outstanding performance. After incorporating SMOTE, the new combined model achieved a significant improvement of nearly 6\% over traditional models and around 5\% over its sub-models, showcasing remarkable results.

Motivation & Objective

• Address transaction fraud in online payment systems with advanced ML models.
• Investigate data preprocessing steps including outlier removal and feature correlation to improve model training.
• Assess the impact of SMOTE on class imbalance for fraud detection.
• Compare gradient-boosting models against traditional classifiers across standard metrics.

Proposed method

• Use LightGBM, XGBoost, and CatBoost as primary models for fraud detection.
• Apply SMOTE to balance imbalanced fraud datasets and evaluate with/without SMOTE.
• Perform data cleaning and outlier removal on selected features (V14, V12, V10).
• Analyze feature correlations and employ dimensionality reduction (t-SNE) for visualization.
• Evaluate models using Precision, Recall, F1 Score, and ROC AUC.
• Present a comparative table of model performance across metrics.

Experimental results

Research questions

• RQ1 Which combination of gradient-boosting models and SMOTE yields the best fraud-detection performance?
• RQ2 How do data preprocessing steps (outlier removal, feature correlation) influence model outcomes?
• RQ3 Do advanced models outperform traditional classifiers (KNN, SVM, Logistic Regression, etc.) in payment fraud detection?
• RQ4 What are the relative strengths of XGBoost, LightGBM, and CatBoost across precision, recall, F1, and AUC?
• RQ5 Is SMOTE beneficial for all models, or are some models (e.g., LightGBM) less sensitive to resampling?

Key findings

• XGBoost+SMOTE achieves the highest combined metrics with precision 0.9999, recall 0.94, F1 0.969, and AUC 0.970.
• XGBoost without SMOTE already performs well (Precision 0.9894, Recall 0.93, F1 0.958, AUC 0.9587).
• CatBoost and LightGBM perform very strong on several metrics, with CatBoost achieving 0.9999 precision and 0.958 recall (F1 0.92; AUC 0.96).
• LightGBM performs exceptionally well even without SMOTE, but SMOTE notably improves precision in several models.
• Compared to traditional models (KNN, SVM, Logistic Regression, Decision Tree, NN), gradient-boosting models show substantial predictive advantages for fraud detection.
• The study concludes that XGBoost+SMOTE is a leading approach for robust anti-fraud payment systems.