[论文解读] An Empirical Study on the Joint Impact of Feature Selection and Data Re-sampling on Imbalance Classification

Chongsheng Zhang, Paolo Soda+4 more|arXiv (Cornell University)|Sep 1, 2021

Imbalanced Data Classification Techniques被引用 4

一句话总结

本文对两类不平衡分类中特征选择与数据重采样联合影响进行了全面的实证研究。在52个数据集、9种特征选择方法、6种重采样技术及3种分类器下，对比了先特征选择后重采样（FS+DS）与先重采样后特征选择（DS+FS）两种流程，结果表明两种流程并无一致的性能优势，应同时评估以实现最优性能。

ABSTRACT

In predictive tasks, real-world datasets often present different degrees of imbalanced (i.e., long-tailed or skewed) distributions. While the majority (the head) classes have sufficient samples, the minority (the tail) classes can be under-represented by a rather limited number of samples. Data pre-processing has been shown to be very effective in dealing with such problems. On one hand, data re-sampling is a common approach to tackling class imbalance. On the other hand, dimension reduction, which reduces the feature space, is a conventional technique for reducing noise and inconsistencies in a dataset. However, the possible synergy between feature selection and data re-sampling for high-performance imbalance classification has rarely been investigated before. To address this issue, we carry out a comprehensive empirical study on the joint influence of feature selection and re-sampling on two-class imbalance classification. Specifically, we study the performance of two opposite pipelines for imbalance classification by applying feature selection before or after data re-sampling. We conduct a large number of experiments, with a total of 9225 tests, on 52 publicly available datasets, using 9 feature selection methods, 6 re-sampling approaches for class imbalance learning, and 3 well-known classification algorithms. Experimental results show that there is no constant winner between the two pipelines; thus both of them should be considered to derive the best performing model for imbalance classification. We find that the performance of an imbalance classification model not only depends on the classifier adopted and the ratio between the number of majority and minority samples, but also depends on the ratio between the number of samples and features. Overall, this study should provide new reference value for researchers and practitioners in imbalance learning.

研究动机与目标

• 探究特征选择与数据重采样对不平衡分类性能的联合影响。
• 比较两种处理流程的有效性：先特征选择后重采样（FS+DS）与先重采样后特征选择（DS+FS）。
• 基于数据集特征与算法选择，识别一种流程优于另一种的条件。
• 为选择最优的特征选择与重采样方法组合，提供实用且基于数据的指导。
• 解决在类别不平衡学习中，特征选择与重采样协同效应缺乏全面实证分析的问题。

提出的方法

• 在52个公开可用的不平衡数据集上执行了9,225次实验。
• 评估了9种特征选择方法、6种重采样技术（包括SMOTE、RUS及组合方法）以及3种分类器（C4.5、SVM、MLP）。
• 应用了两种处理流程：(1) 先特征选择后重采样（FS+DS），(2) 先重采样后特征选择（DS+FS）。
• 使用多种评估指标：G-Mean、F1-score与IBA（改进的平衡准确率），并通过Iman-Davenport检验进行统计显著性分析。
• 采用基于Rank-SUM的启发式Top-K排序方法，识别在各数据集上表现最优的方法组合。
• 分析了不平衡比（IR）与样本-特征比（SFR）对流程性能的影响。

实验结果

研究问题

• RQ1 在多样化数据集上，FS+DS与DS+FS两种流程中，哪一种在二分类不平衡分类中表现更优？
• RQ2 基础分类器选择（C4.5、SVM、MLP）、重采样方法与特征选择技术的选择如何影响流程性能？
• RQ3 数据集不平衡比（IR）与样本-特征比（SFR）如何影响两种流程相对有效性的差异？
• RQ4 是否存在某些特征选择与重采样方法的组合，在所有评估指标上均持续优于其他组合？
• RQ5 特征选择与重采样顺序对不平衡学习中模型泛化能力与鲁棒性的影响程度如何？

主要发现

• FS+DS与DS+FS两种流程并无一致的胜出者；为实现最优性能，应同时评估两种流程。
• 当使用C4.5作为基础分类器时，DS+FS流程在G-Mean与F1-score指标上优于FS+DS，尤其是在使用过采样方法时。
• 当使用SVM作为基础分类器时，使用欠采样方法时FS+DS优于DS+FS，但使用过采样方法时无明显胜出者。
• 对于不平衡比（IR）高且样本-特征比（SFR）低的数据集，FS+DS流程表现出相对优势。
• 对于SFR高且IR低的数据集，无任一流程持续占优，表明性能取决于具体的数据集与方法组合。
• 表现最优的方法组合——如DS+FS下的SMOTE+RS与Linear+RUS，以及FS+DS下的ChiS+RUS与Ttest+RUS——在统计检验（Iman-Davenport）与启发式评估（Top-K）中均稳定位列前三。