[Paper Review] Survey of resampling techniques for improving classification performance in unbalanced datasets

Ajinkya More|arXiv (Cornell University)|Aug 22, 2016

Imbalanced Data Classification TechniquesComputer Science10 references179 citations

TL;DR

The paper reviews resampling techniques for imbalanced classification, comparing many methods on a synthetic dataset with respect to minority recall and majority precision.

ABSTRACT

A number of classification problems need to deal with data imbalance between classes. Often it is desired to have a high recall on the minority class while maintaining a high precision on the majority class. In this paper, we review a number of resampling techniques proposed in literature to handle unbalanced datasets and study their effect on classification performance.

Motivation & Objective

• Motivate and define data imbalance problems across domains such as fraud detection, product categorization, and disease diagnosis.
• Systematically compare a wide range of resampling techniques on a synthetic two-class dataset to assess impact on minority recall and majority precision.
• Demonstrate how different methods affect decision boundaries when using common classifiers like logistic regression.
• Highlight combinations and ensembles that offer best performance under chosen metrics within this study.

Proposed method

• Use a synthetic two-class dataset with skewed class weights to simulate imbalance (r = |S|/|L| = 0.1).
• Evaluate baseline logistic regression with standard cross validation to establish a reference.
• Apply class-weighted loss, undersampling (random, NearMiss variants, CNN, ENN, Tomek), oversampling (random, SMOTE, Borderline-SMOTE variants), and combinations (SMOTE+ENN, SMOTE+Tomek).
• Test ensemble approaches (EasyEnsemble, BalanceCascade) that combine resampling with boosting.
• Report performance in terms of precision on the majority class (L) and recall on the minority class (S) on a 70/30 train/test split with 5-fold CV for parameter selection.

Experimental results

Research questions

• RQ1 How do different resampling strategies affect the trade-off between precision on the majority class and recall on the minority class?
• RQ2 Which resampling method or combination yields the best minority recall while maintaining high majority precision on synthetic imbalanced data?
• RQ3 Do ensemble methods outperform single-model resampling approaches on the given metrics?
• RQ4 What is the impact of using weighted loss versus resampling alone on classification performance?
• RQ5 Are there particular variants (e.g., Borderline-SMOTE, ENN, Tomek, SMOTE+ENN) that consistently perform better across this setup?

Key findings

• Baseline logistic regression with no resampling yields low minority recall (0.12) with high majority precision (0.90).
• Weighted loss improves minority recall (0.89) with high majority precision (0.98).
• Undersampling methods generally reduce major class precision but can improve minority recall in some variants (e.g., NearMiss variants, CNN, ENN) with varying effects.
• Oversampling methods, especially SMOTE variants and combinations (SMOTE+ENN, SMOTE+Tomek), markedly improve minority recall with maintained or high majority precision.
• Ensemble methods like EasyEnsemble and BalanceCascade achieve high precision on L (≈0.99) and minority recall (≈0.91).
• Overall, on the synthetic dataset, SMOTE+ENN with logistic regression and BalanceCascade yielded strong performance for the chosen metrics.