[Paper Review] Detecting Anti-Semitic Hate Speech using Transformer-based Large Language Models

Dengyi Liu, Minghao Wang+1 more|arXiv (Cornell University)|May 6, 2024

Hate Speech and Cyberbullying Detection4 citations

TL;DR

This paper proposes a transformer-based approach for detecting anti-Semitic hate speech using fine-tuned BERT, RoBERTa, and LLaMA-2 models with LoRA adaptation. It demonstrates that transformer models, especially BERT and RoBERTa, significantly outperform traditional machine learning methods in precision, recall, and F1-score, with LoRA fine-tuning further improving detection sensitivity for subtle hate speech patterns.

ABSTRACT

Academic researchers and social media entities grappling with the identification of hate speech face significant challenges, primarily due to the vast scale of data and the dynamic nature of hate speech. Given the ethical and practical limitations of large predictive models like ChatGPT in directly addressing such sensitive issues, our research has explored alternative advanced transformer-based and generative AI technologies since 2019. Specifically, we developed a new data labeling technique and established a proof of concept targeting anti-Semitic hate speech, utilizing a variety of transformer models such as BERT (arXiv:1810.04805), DistillBERT (arXiv:1910.01108), RoBERTa (arXiv:1907.11692), and LLaMA-2 (arXiv:2307.09288), complemented by the LoRA fine-tuning approach (arXiv:2106.09685). This paper delineates and evaluates the comparative efficacy of these cutting-edge methods in tackling the intricacies of hate speech detection, highlighting the need for responsible and carefully managed AI applications within sensitive contexts.

Motivation & Objective

• To address the growing challenge of anti-Semitic hate speech on social media platforms like Twitter, where traditional detection methods often fail due to linguistic nuance and context.
• To develop a novel data labeling technique that enables democratic, consensus-based annotation of anti-Semitic content, improving dataset quality and reliability.
• To evaluate the comparative performance of traditional machine learning models (e.g., SVM, Naive Bayes, Random Forest) versus state-of-the-art transformer-based models (e.g., BERT, RoBERTa, LLaMA-2) in detecting anti-Semitic hate speech.
• To investigate the impact of LoRA fine-tuning on large language models like RoBERTa and LLaMA-2 (7B) in improving detection performance, particularly in recall and F1-score.
• To contribute to safer online communication by advancing automated content moderation with interpretable, high-accuracy NLP models for sensitive hate speech detection.

Proposed method

• Employed a custom data labeling algorithm based on democratic consensus to annotate anti-Semitic content from Twitter, ensuring high-quality, context-aware labeling.
• Utilized multiple embedding techniques including CountVectorizer, TfidfVectorizer, and HashingVectorizer for traditional machine learning models, and contextual embeddings from BERT, DistilBERT, RoBERTa, and LLaMA-2 for deep learning models.
• Applied traditional classifiers such as k-NN, logistic regression, SVM, random forest, and naive Bayes to baseline models using extracted features.
• Fine-tuned RoBERTa and LLaMA-2 (7B) using Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method that preserves pre-trained knowledge while adapting to hate speech detection.
• Evaluated model performance using standard NLP metrics: accuracy, precision, recall, and F1-score across all models on the annotated dataset.
• Compared the performance of traditional models with transformer-based models to assess the impact of contextual understanding and attention mechanisms on hate speech detection.

Figure 1: This figure shows the workflow of voting algorithm.

Experimental results

Research questions

• RQ1 How do transformer-based models like BERT, RoBERTa, and LLaMA-2 compare to traditional machine learning models in detecting anti-Semitic hate speech?
• RQ2 To what extent does LoRA fine-tuning improve the performance of large language models such as RoBERTa and LLaMA-2 (7B) in detecting subtle or contextually embedded anti-Semitic content?
• RQ3 How does the choice of embedding technique (e.g., TF-IDF, word embeddings, contextual embeddings) affect the performance of hate speech detection models?
• RQ4 What is the impact of model architecture and computational efficiency on the detection of anti-Semitic hate speech, particularly in terms of recall and F1-score?
• RQ5 Can a consensus-based data labeling approach produce a reliable and representative dataset for training high-accuracy hate speech detection systems?

Key findings

• BERT achieved the highest accuracy and F1-score among all models evaluated, demonstrating superior performance in detecting anti-Semitic hate speech due to its deep contextual understanding.
• RoBERTa and LLaMA-2 (7B) showed significant improvements in recall and F1-score after LoRA fine-tuning, indicating enhanced sensitivity to nuanced and contextually embedded hate speech.
• Traditional models like k-NN performed the worst across all metrics, highlighting limitations in handling high-dimensional and sparse textual data common in hate speech detection.
• While SVM and logistic regression offered decent baseline performance, they consistently underperformed compared to transformer models, especially in recall, indicating failure to detect many relevant cases.
• The combination of contextual embeddings (e.g., from RoBERTa) and LoRA fine-tuning led to a notable 10–15% improvement in F1-score for LLaMA-2 (7B), suggesting that parameter-efficient fine-tuning enhances model adaptability.
• The study confirms that transformer-based models are significantly more effective than traditional machine learning models in capturing linguistic complexity and context in anti-Semitic hate speech.

Figure 2: Architecture of LoRA fine-tuning applied to Llama2 model weights, demonstrating how the injection of trainable low-rank matrices A and B (initialized as A with a normal distribution and B set to zero) allows for efficient adaptation of the model’s weights to new tasks, preserving the origi