[Paper Review] Building an Effective Email Spam Classification Model with spaCy

Kazem Taghandiki|arXiv (Cornell University)|Mar 15, 2023

Spam and Phishing Detection4 citations

TL;DR

This paper proposes a spam email detection system using spaCy for NLP preprocessing and three machine learning models—Naive Bayes, Decision Tree C45, and Multilayer Perceptron (MLP)—on a dataset of 1,500 Gmail emails. The MLP model achieved the highest accuracy of 96%, demonstrating its effectiveness in classifying spam versus legitimate emails with high precision (97%) and F1-score (96%).

ABSTRACT

Today, people use email services such as Gmail, Outlook, AOL Mail, etc. to communicate with each other as quickly as possible to send information and official letters. Spam or junk mail is a major challenge to this type of communication, usually sent by botnets with the aim of advertising, harming and stealing information in bulk to different people. Receiving unwanted spam emails on a daily basis fills up the inbox folder. Therefore, spam detection is a fundamental challenge, so far many works have been done to detect spam using clustering and text categorisation methods. In this article, the author has used the spaCy natural language processing library and 3 machine learning (ML) algorithms Naive Bayes (NB), Decision Tree C45 and Multilayer Perceptron (MLP) in the Python programming language to detect spam emails collected from the Gmail service. Observations show the accuracy rate (96%) of the Multilayer Perceptron (MLP) algorithm in spam detection.

Motivation & Objective

• To address the growing challenge of spam emails that compromise user security and inbox integrity.
• To develop a robust text classification pipeline using modern NLP techniques for effective spam detection.
• To evaluate and compare the performance of Naive Bayes, Decision Tree C45, and Multilayer Perceptron on a real-world email dataset.
• To demonstrate the effectiveness of spaCy in preprocessing email text for downstream machine learning tasks.
• To provide a high-accuracy, scalable solution for automated spam filtering in email systems.

Proposed method

• Collected 1,500 email samples (1,125 for training, 375 for testing) from Gmail, labeled as spam or legitimate.
• Applied spaCy for text preprocessing, including tokenization, lemmatization, stopword removal, and POS tagging.
• Utilized scikit-learn implementations of Naive Bayes, Decision Tree C45, and Multilayer Perceptron for model training.
• Trained models on preprocessed text features extracted from email content using bag-of-words or similar vectorization.
• Evaluated models using standard metrics: accuracy, precision, recall, and F1-score on the test set.
• Employed confusion matrices and performance visualization to compare model outcomes.

Figure 1: Implementation process of the proposed approach

Experimental results

Research questions

• RQ1 Can spaCy-based text preprocessing significantly improve the performance of machine learning models in spam email classification?
• RQ2 How do Naive Bayes, Decision Tree C45, and Multilayer Perceptron compare in classifying spam versus legitimate emails on a real Gmail dataset?
• RQ3 What is the optimal balance between precision, recall, and accuracy in spam detection using traditional machine learning models?
• RQ4 To what extent does the choice of machine learning algorithm impact the detection of malicious or deceptive email content?
• RQ5 Can a Multilayer Perceptron outperform simpler models like Naive Bayes and Decision Trees in this specific spam classification task?

Key findings

• The Multilayer Perceptron (MLP) model achieved the highest accuracy of 96% in classifying spam emails.
• The MLP model recorded a precision of 97% and an F1-score of 96%, indicating strong performance in minimizing false positives and balancing precision and recall.
• The Decision Tree C45 model achieved the highest recall (95%), suggesting better detection of actual spam instances compared to the other models.
• Naive Bayes showed moderate performance with lower accuracy and F1-score compared to MLP, indicating limitations in capturing complex patterns in email text.
• The use of spaCy for preprocessing—including lemmatization, stopword removal, and tokenization—significantly contributed to improved model performance across all algorithms.
• The confusion matrices confirmed that MLP had the lowest number of misclassified instances, particularly reducing false negatives.

Figure 2: Tokenisation pre-processing