[Paper Review] Improving spam filtering by combining Naive Bayes with simple k-nearest neighbor searches

Daniel Etzold|ArXiv.org|Nov 30, 2003

Spam and Phishing Detection4 references3 citations

TL;DR

This paper proposes a hybrid spam filtering approach combining Naive Bayes with k-nearest neighbor (kNN) search using Euclidean distance or cosine angle similarity. By fusing probabilistic scores from both models with equal weighting, the method improves classification accuracy—especially for legitimate emails—while reducing feature dimensionality from 2000 to 500 features, achieving up to 99.25% accuracy on legitimate emails with k=1 and V=2000.

ABSTRACT

Using naive Bayes for email classification has become very popular within the last few months. They are quite easy to implement and very efficient. In this paper we want to present empirical results of email classification using a combination of naive Bayes and k-nearest neighbor searches. Using this technique we show that the accuracy of a Bayes filter can be improved slightly for a high number of features and significantly for a small number of features.

Motivation & Objective

• To improve spam filtering accuracy beyond standard Naive Bayes by integrating k-nearest neighbor classification.
• To reduce the number of features required for high accuracy, lowering computational cost.
• To minimize false positives in legitimate email classification, which are more costly than false negatives.
• To evaluate the effectiveness of combining Naive Bayes with kNN using both Euclidean distance and cosine angle as similarity measures.

Proposed method

• Combines Naive Bayes probabilities with kNN probabilities using a weighted average score δ = (αPrnb + βPrknn)/(α + β), with α = β = 1.
• Uses Euclidean distance and cosine angle between document vectors as similarity measures for kNN.
• Applies information gain to reduce feature dimensions to 500, 1000, 1500, and 2000 features.
• Tokenizes emails by lowercasing, removing HTML tags, extracting words and numbers with ≥2 characters, and appending 'html' for each HTML tag.
• Classifies emails based on the higher δ score: δG ≥ 0.5 for 'legitimate', otherwise 'spam'.
• Employs kNN with k ∈ {1, 2, 3, 4, 5} and evaluates performance across multiple training/test splits (25:75, 30:70, 40:60).

Experimental results

Research questions

• RQ1 Can combining Naive Bayes with kNN improve spam filtering accuracy compared to Naive Bayes alone?
• RQ2 Does the hybrid approach allow for significant reduction in feature dimensionality without sacrificing accuracy?
• RQ3 Which similarity measure—Euclidean distance or cosine angle—yields better classification performance?
• RQ4 What is the optimal k value for kNN in the hybrid model, and how does it vary between spam and legitimate email classification?
• RQ5 Can the hybrid model significantly reduce false positives in legitimate email classification?

Key findings

• The hybrid model achieves 99.25% accuracy in classifying legitimate emails when using cosine angle similarity with V=2000 and k=1, surpassing Naive Bayes alone.
• For V=500 features, the hybrid model achieves 99.10% accuracy on legitimate emails with k=1, exceeding the 98.65% accuracy of Naive Bayes with V=2000.
• The number of false positives is dramatically reduced, especially when k=1, making the approach highly effective for minimizing false positives.
• For spam classification, the best accuracy with low-dimensional features (V=500) is achieved at k=2, outperforming Naive Bayes with V=2000.
• The cosine angle similarity measure performs comparably to Euclidean distance, with both yielding high accuracy and robustness across feature dimensions.
• Using only 500 features with k=1 or k=2 achieves higher accuracy than Naive Bayes with 2000 features, significantly reducing computational cost while improving performance.