[Paper Review] Learning to Filter Spam E-Mail: A Comparison of a Naive Bayesian and a Memory-Based Approach

Ion Androutsopoulos, Γεώργιος Παλιούρας+4 more|ArXiv.org|Sep 18, 2000

Spam and Phishing Detection17 references282 citations

TL;DR

This paper compares Naive Bayesian and memory-based learning approaches for spam email filtering using a public email corpus. Both methods significantly outperform keyword-based filters, with the Naive Bayes classifier achieving high accuracy and the memory-based approach showing strong performance under cost-sensitive evaluation, establishing benchmarks for anti-spam filtering research.

ABSTRACT

We investigate the performance of two machine learning algorithms in the context of anti-spam filtering. The increasing volume of unsolicited bulk e-mail (spam) has generated a need for reliable anti-spam filters. Filters of this type have so far been based mostly on keyword patterns that are constructed by hand and perform poorly. The Naive Bayesian classifier has recently been suggested as an effective method to construct automatically anti-spam filters with superior performance. We investigate thoroughly the performance of the Naive Bayesian filter on a publicly available corpus, contributing towards standard benchmarks. At the same time, we compare the performance of the Naive Bayesian filter to an alternative memory-based learning approach, after introducing suitable cost-sensitive evaluation measures. Both methods achieve very accurate spam filtering, outperforming clearly the keyword-based filter of a widely used e-mail reader.

Motivation & Objective

• To evaluate and compare the performance of Naive Bayesian and memory-based learning algorithms in spam email filtering.
• To establish a standard benchmark for anti-spam filtering using a publicly available email corpus.
• To assess the effectiveness of machine learning methods compared to traditional keyword-based filtering techniques.
• To introduce and apply cost-sensitive evaluation measures to better reflect real-world filtering trade-offs.
• To provide empirical evidence on the superiority of learning-based filters over handcrafted keyword rules.

Proposed method

• The Naive Bayesian classifier is applied to a public email corpus, modeling spam detection as a binary classification problem based on word frequencies.
• The memory-based learning approach uses case-based reasoning, storing and retrieving past email examples based on similarity to new inputs.
• Both models are trained and tested on the same publicly available email dataset to ensure fair comparison.
• Cost-sensitive evaluation measures are introduced to account for the asymmetric costs of false positives and false negatives in spam filtering.
• Performance is evaluated using standard metrics such as precision, recall, and F1-score, with emphasis on minimizing false positives.
• The models are compared under identical conditions to isolate the impact of the learning algorithm.

Experimental results

Research questions

• RQ1 How do Naive Bayesian and memory-based learning methods compare in terms of spam filtering accuracy on a public email corpus?
• RQ2 To what extent do learning-based filters outperform traditional keyword-based filters in spam detection?
• RQ3 How do cost-sensitive evaluation measures affect the performance ranking of the two learning algorithms?
• RQ4 Can the Naive Bayesian classifier serve as a reliable benchmark for future anti-spam filtering research?
• RQ5 What are the relative strengths and weaknesses of each approach in handling real-world email filtering trade-offs?

Key findings

• The Naive Bayesian classifier achieved high accuracy in spam detection, significantly outperforming the keyword-based filter in the same evaluation.
• The memory-based learning approach also demonstrated strong performance, approaching or matching the Naive Bayes results under cost-sensitive evaluation.
• Both learning-based methods substantially reduced false positive rates compared to the keyword-based filter, which is critical for user acceptance.
• The study confirms that machine learning-based filters are more effective and reliable than handcrafted keyword rules for spam detection.
• The public email corpus used in the study is established as a valid benchmark for future anti-spam filtering research.
• Cost-sensitive evaluation revealed nuanced differences in model behavior, highlighting the importance of evaluation metrics aligned with real-world priorities.