[Paper Review] A Comparison Study of Credit Card Fraud Detection: Supervised versus Unsupervised

Xuetong Niu, Li Wang+1 more|arXiv (Cornell University)|Apr 24, 2019

Imbalanced Data Classification Techniques22 references72 citations

TL;DR

The paper compares 6 supervised and 4 unsupervised credit card fraud detection models on a Kaggle dataset, using AUROC with 5-fold cross-validation, finding supervised models slightly outperform unsupervised ones overall.

ABSTRACT

Credit card has become popular mode of payment for both online and offline purchase, which leads to increasing daily fraud transactions. An Efficient fraud detection methodology is therefore essential to maintain the reliability of the payment system. In this study, we perform a comparison study of credit card fraud detection by using various supervised and unsupervised approaches. Specifically, 6 supervised classification models, i.e., Logistic Regression (LR), K-Nearest Neighbors (KNN), Support Vector Machines (SVM), Decision Tree (DT), Random Forest (RF), Extreme Gradient Boosting (XGB), as well as 4 unsupervised anomaly detection models, i.e., One-Class SVM (OCSVM), Auto-Encoder (AE), Restricted Boltzmann Machine (RBM), and Generative Adversarial Networks (GAN), are explored in this study. We train all these models on a public credit card transaction dataset from Kaggle website, which contains 492 frauds out of 284,807 transactions. The labels of the transactions are used for supervised learning models only. The performance of each model is evaluated through 5-fold cross validation in terms of Area Under the Receiver Operating Curves (AUROC). Within supervised approaches, XGB and RF obtain the best performance with AUROC = 0.989 and AUROC = 0.988, respectively. While for unsupervised approaches, RBM achieves the best performance with AUROC = 0.961, followed by GAN with AUROC = 0.954. The experimental results show that supervised models perform slightly better than unsupervised models in this study. Anyway, unsupervised approaches are still promising for credit card fraud transaction detection due to the insufficient annotation and the data imbalance issue in real-world applications.

Motivation & Objective

• Assess and compare the performance of supervised and unsupervised machine learning models for credit card fraud detection.
• Evaluate how data labeling, imbalance, and annotation latency affect model performance.
• Identify which model families (supervised vs unsupervised) yield higher AUROC on a real-world fraud dataset.

Proposed method

• Evaluate 6 supervised models (LR, KNN, SVM, DT, RF, XGB) using labeled fraud data with downsampling to balance classes.
• Evaluate 4 unsupervised models (OCSVM, AE, RBM, GAN) trained on unlabeled data to detect anomalies.
• Use 5-fold cross-validation and AUROC as the performance metric.
• Normalize Time and Amount with RobustScaler and downsample non-fraud cases to equal fraud cases (492 each).
• Tune hyperparameters with grid-search within cross-validation.
• Provide implementation details for AE and GAN architectures used.

Experimental results

Research questions

• RQ1 How do supervised models compare to unsupervised models in AUROC for credit card fraud detection on a real, highly imbalanced dataset?
• RQ2 What are the top-performing models within each category (supervised vs unsupervised) in this study?
• RQ3 What are the practical trade-offs between supervised and unsupervised approaches given labeling requirements and data imbalance?

Key findings

• XGBoost (supervised) achieves the highest AUROC of 0.989–0.990 on the dataset.
• Random Forest (supervised) achieves AUROC of 0.988.
• Decision Tree (supervised) achieves AUROC of 0.95 (the lowest among supervised models).
• RBM (unsupervised) achieves AUROC of 0.961, making it the best among unsupervised methods in this study.
• GAN (unsupervised) achieves AUROC of 0.954.
• OC-SVM (unsupervised) achieves AUROC of 0.90, the lowest among the unsupervised models.