[Paper Review] Phishing Detection through Email Embeddings

Luis Felipe González Gutiérrez, Faranak Abri+3 more|arXiv (Cornell University)|Dec 28, 2020

Spam and Phishing Detection21 references4 citations

TL;DR

This paper proposes using doc2vec-based email embeddings to detect phishing emails by capturing semantic content, even when phishing and legitimate emails share similar surface-level features. Despite high similarity in textual indicators, the method achieves 91.6% accuracy and 90.0% F1-score using Random Forest on a 2D linear PCA projection of embeddings, demonstrating that semantic vectorization effectively distinguishes phishing from legitimate emails.

ABSTRACT

The problem of detecting phishing emails through machine learning techniques has been discussed extensively in the literature. Conventional and state-of-the-art machine learning algorithms have demonstrated the possibility of building classifiers with high accuracy. The existing research studies treat phishing and genuine emails through general indicators and thus it is not exactly clear what phishing features are contributing to variations of the classifiers. In this paper, we crafted a set of phishing and legitimate emails with similar indicators in order to investigate whether these cues are captured or disregarded by email embeddings, i.e., vectorizations. We then fed machine learning classifiers with the carefully crafted emails to find out about the performance of email embeddings developed. Our results show that using these indicators, email embeddings techniques is effective for classifying emails as phishing or legitimate.

Motivation & Objective

• To investigate whether machine learning models trained on email embeddings can distinguish phishing from legitimate emails when textual features are nearly identical.
• To evaluate the effectiveness of doc2vec embeddings in capturing semantic differences despite superficial similarity in phishing and legitimate email content.
• To determine whether email embeddings preserve discriminative features that enable high-accuracy classification even when conventional features are indistinguishable.
• To compare the performance of multiple classifiers (SVM, Logistic Regression, Random Forest, Naive Bayes) on both full embedding space and low-dimensional projections.
• To analyze the underlying structure of email embeddings using explained variance and dimensionality reduction techniques (linear and RBF kernel PCA).

Proposed method

• Constructed a dataset of 12 phishing and 12 legitimate emails with deliberately similar linguistic and structural indicators to isolate semantic differences.
• Applied doc2vec to convert each email into a 20-dimensional dense vector representation that captures semantic meaning.
• Trained multiple binary classifiers—SVM, Logistic Regression, Random Forest, and Naive Bayes—on the full 20D embedding space.
• Performed dimensionality reduction using linear PCA and RBF kernel PCA to project embeddings into 2D for visualization and analysis.
• Evaluated model performance using accuracy and F1-score across different embedding representations and classifier types.
• Conducted explained variance analysis to determine how much of the total variance in embeddings is captured by the first few principal components.

Experimental results

Research questions

• RQ1 Can email embeddings effectively distinguish phishing from legitimate emails when the two types share nearly identical surface-level features?
• RQ2 How do different machine learning classifiers perform when trained on doc2vec-generated email embeddings?
• RQ3 Does the underlying structure of email embeddings support linear or non-linear separability in the semantic space?
• RQ4 What proportion of variance in the embedding space is sufficient to achieve high classification accuracy?
• RQ5 To what extent do 2D projections of embeddings preserve the discriminative power of the original 20D feature space?

Key findings

• The Random Forest classifier achieved the highest accuracy of 91.6% and F1-score of 90.0% when using a 2D linear PCA projection of the embeddings, despite using only 25% of the original variance.
• SVM achieved the best performance in the original 20D embedding space with 81.6% accuracy and 76.6% F1-score.
• The 2D linear PCA projection outperformed both the original 20D space and the RBF kernel PCA projection, suggesting the underlying embedding structure is approximately linear.
• The RBF kernel PCA projection yielded slightly lower performance (78.3% accuracy, 76.6% F1-score), indicating less effective class segmentation in this nonlinear projection.
• The explained variance analysis showed that 90% of the total variance is captured by the first 12 principal components, highlighting the low intrinsic dimensionality of the embedding space.
• The consistent high performance across multiple classifiers and projections indicates that doc2vec embeddings effectively encode semantic distinctions between phishing and legitimate emails.