[论文解读] Spam filtering on forums: A synthetic oversampling based approach for imbalanced data classification

Pratik Ratadiya, Rahul Moorthy|arXiv (Cornell University)|Sep 10, 2019

Spam and Phishing Detection参考文献 1被引用 12

一句话总结

本文提出一种基于SMOTE的合成过采样方法，以解决论坛垃圾信息过滤中的类别不平衡问题，显著提升了分类性能。通过生成少数类（垃圾信息）的合成样本，该方法在多种算法上均提升了模型的召回率和F1分数，相较于在不平衡数据上训练的基线模型，F1分数最高提升达10%。

ABSTRACT

Forums play an important role in providing a platform for community interaction. The introduction of irrelevant content or spam by individuals for commercial and social gains tends to degrade the professional experience presented to the forum users. Automated moderation of the relevancy of posted content is desired. Machine learning is used for text classification and finds applications in spam email detection, fraudulent transaction detection etc. The balance of classes in training data is essential in the case of classification algorithms to make the learning efficient and accurate. However, in the case of forums, the spam content is sparse compared to the relevant content giving rise to a bias towards the latter while training. A model trained on such biased data will fail to classify a spam sample. An approach based on Synthetic Minority Over-sampling Technique(SMOTE) is presented in this paper to tackle imbalanced training data. It involves synthetically creating new minority class samples from the existing ones until balance in data is achieved. The enhanced data is then passed through various classifiers for which the performance is recorded. The results were analyzed on the data of forums of Spoken Tutorial, IIT Bombay over standard performance metrics and revealed that models trained after Synthetic Minority oversampling outperform the ones trained on imbalanced data by substantial margins. An empirical comparison of the results obtained by both SMOTE and without SMOTE for various supervised classification algorithms have been presented in this paper. Synthetic oversampling proves to be a critical technique for achieving uniform class distribution which in turn yields commendable results in text classification. The presented approach can be further extended to content categorization on educational websites thus helping to improve the overall digital learning experience.

研究动机与目标

• 为解决论坛垃圾信息检测中垃圾信息实例远少于非垃圾内容的类别不平衡问题。
• 通过平衡少数类（垃圾信息）和多数类（非垃圾信息）来提升监督分类算法在不平衡文本数据上的性能。
• 评估通过SMOTE进行合成过采样在提升教育论坛垃圾信息过滤中精确率、召回率和F1分数方面的有效性。
• 证明基于SMOTE的数据增强相比在原始不平衡数据集上训练，能带来更稳健和准确的垃圾信息分类结果。

提出的方法

• 预处理步骤包括去除HTML标签、删除特殊字符和停用词，并过滤掉长度少于3个字符的词元。
• 使用TF-IDF对文本数据进行向量化，通过计算词频-逆文档频率权重来表示文本特征。
• 在训练集上应用SMOTE，以生成合成的垃圾信息样本，从而平衡少数类（垃圾信息）和多数类（非垃圾信息）。
• 将过采样后的数据集用于训练多种监督分类器：多项式朴素贝叶斯、逻辑回归、线性SVC和决策树。
• 通过精确率、召回率、F1分数和准确率评估性能，重点关注对少数类检测敏感的指标。
• 将SMOTE过采样数据与原始不平衡数据集在所有算法上进行对比，以评估性能提升。

实验结果

研究问题

• RQ1 使用SMOTE进行合成过采样是否能显著提升在不平衡论坛数据集中的垃圾信息检测效果？
• RQ2 基于SMOTE的数据平衡对各种文本分类算法在垃圾信息过滤中的精确率、召回率和F1分数有何影响？
• RQ3 SMOTE能否缓解在真实论坛数据上训练的垃圾信息检测模型对多数类的偏差？
• RQ4 在不平衡论坛文本数据上，哪种分类算法在F1分数提升方面从SMOTE中获益最多？

主要发现

• 应用SMOTE后，多项式朴素贝叶斯的F1分数从0.71提升至0.82，较基线模型提升10%。
• 对于逻辑回归，应用SMOTE后F1分数从0.0提升至0.40，表明在少数类检测方面取得显著进步。
• 线性SVC模型的F1分数在应用SMOTE后从0.0提升至0.60，显示出对召回率和精确率的显著积极影响。
• 决策树模型的F1分数从0.20提升至0.18，提升幅度较小，且仍为最弱表现者，表明在此情况下SMOTE带来的收益有限。
• 整体准确率保持较高水平（SMOTE下为0.96），但真正的提升体现在召回率和F1分数，这对不平衡垃圾信息检测更具实际意义。
• 本研究证实，SMOTE能有效减少对多数类的偏差，并增强模型检测稀有垃圾信息实例的能力。