[Paper Review] Boosting Trees for Anti-Spam Email Filtering

Xavier Carreras, Lluı́s Màrquez|ArXiv.org|Sep 13, 2001

Spam and Phishing Detection15 references342 citations

TL;DR

This paper evaluates AdaBoost with confidence-rated predictions for anti-spam email filtering on the PU1 corpus, demonstrating that boosting-based classifiers significantly outperform Naive Bayes and decision trees, achieving F1 scores above 97%. It further shows that increasing base learner complexity improves high-precision filtering, crucial for minimizing false positives in real-world deployment.

ABSTRACT

This paper describes a set of comparative experiments for the problem of automatically filtering unwanted electronic mail messages. Several variants of the AdaBoost algorithm with confidence-rated predictions [Schapire & Singer, 99] have been applied, which differ in the complexity of the base learners considered. Two main conclusions can be drawn from our experiments: a) The boosting-based methods clearly outperform the baseline learning algorithms (Naive Bayes and Induction of Decision Trees) on the PU1 corpus, achieving very high levels of the F1 measure; b) Increasing the complexity of the base learners allows to obtain better ``high-precision'' classifiers, which is a very important issue when misclassification costs are considered.

Motivation & Objective

• To evaluate the effectiveness of AdaBoost with confidence-rated predictions for spam email filtering compared to baseline methods.
• To investigate whether increasing the complexity of base learners improves performance, especially in high-precision scenarios.
• To assess the impact of misclassification costs in real-world spam filtering, where false positives are highly detrimental.
• To provide a robust, overfitting-resistant method that requires no prior feature filtering, even with large feature sets.
• To demonstrate the utility of confidence scores and tunable thresholds for practical deployment in email filtering systems.

Proposed method

• Applies the AdaBoost algorithm with confidence-rated predictions to learn a strong classifier from weak hypotheses.
• Uses decision stumps and increasingly complex decision trees (up to depth 5) as base learners to assess expressiveness impact.
• Employs a weighted combination of weak hypotheses, where each is trained on a distribution over training examples that evolves across rounds.
• Tunes the number of boosting rounds and classification threshold to optimize precision and recall trade-offs.
• Uses cost-sensitive evaluation metrics from Androutsopoulos et al. (2000b) to assess performance under misclassification cost constraints.
• Generates precision-recall curves by varying the decision threshold to analyze performance at high-precision levels (e.g., 95–100%).

Experimental results

Research questions

• RQ1 Does AdaBoost with confidence-rated predictions outperform traditional methods like Naive Bayes and decision trees on the PU1 spam corpus?
• RQ2 How does increasing the complexity of base learners (e.g., from stumps to deeper trees) affect the performance of boosting-based spam filters?
• RQ3 Can AdaBoost produce high-precision classifiers suitable for real-world email filtering where false positives are costly?
• RQ4 How do confidence scores and adjustable decision thresholds improve the practical deployment of spam filters?
• RQ5 To what extent do tuning procedures and model complexity influence the robustness and generalization of the classifier?

Key findings

• AdaBoost with confidence-rated predictions achieved F1 scores above 97% on the PU1 corpus, significantly outperforming Naive Bayes and decision trees.
• The Stumps variant achieved 98.58% F1, while TreeBoost[4] reached 99.14% F1, demonstrating the benefit of more expressive base learners.
• At 100% precision, TreeBoost[4] achieved 80.24% recall, outperforming Stumps (62.37%) and other TreeBoost variants, indicating better high-precision performance.
• All variants achieved 98.75% recall at 95% precision, but deeper trees (e.g., TreeBoost[4]) maintained higher recall at 99% and 100% precision levels.
• The method showed strong resistance to overfitting, even with tens of thousands of features, without requiring prior feature filtering.
• The confidence scores from the model enabled effective threshold tuning, allowing deployment of high-precision filters that minimize false positives.