[Paper Review] Detecting Harmful Online Conversational Content towards LGBTQIA+ Individuals

Jamell Dacon, Harry Shomer+2 more|arXiv (Cornell University)|Jun 15, 2022

Hate Speech and Cyberbullying Detection4 citations

TL;DR

This paper introduces a novel multi-label dataset for detecting anti-LGBTQIA+ conversational content, employing both traditional machine learning (SVM, Logistic Regression) and fine-tuned large language models (BERT, RoBERTa, HateBERT) to classify six harmful language categories. It demonstrates that BERT achieves F1-scores ≥0.82 across most labels, highlighting strong performance in identifying anti-LGBTQIA+ toxicity despite challenges with low-resource threat labels.

ABSTRACT

Online discussions, panels, talk page edits, etc., often contain harmful conversational content i.e., hate speech, death threats and offensive language, especially towards certain demographic groups. For example, individuals who identify as members of the LGBTQIA+ community and/or BIPOC (Black, Indigenous, People of Color) are at higher risk for abuse and harassment online. In this work, we first introduce a real-world dataset that will enable us to study and understand harmful online conversational content. Then, we conduct several exploratory data analysis experiments to gain deeper insights from the dataset. We later describe our approach for detecting harmful online Anti-LGBTQIA+ conversational content, and finally, we implement two baseline machine learning models (i.e., Support Vector Machine and Logistic Regression), and fine-tune 3 pre-trained large language models (BERT, RoBERTa, and HateBERT). Our findings verify that large language models can achieve very promising performance on detecting online Anti-LGBTQIA+ conversational content detection tasks.

Motivation & Objective

• To address the growing prevalence of anti-LGBTQIA+ hate speech, abuse, and offensive language in online conversations, especially on platforms like Reddit.
• To create a high-quality, multi-labeled dataset for automated detection of harmful content targeting LGBTQIA+ individuals across six distinct toxicity categories.
• To evaluate the effectiveness of both traditional machine learning models and fine-tuned large language models in detecting anti-LGBTQIA+ conversational toxicity.
• To support the development of safer, more inclusive online environments by enabling future shared tasks and ethical AI deployment in content moderation.

Proposed method

• Adapted the gender orientation dimension from RedditBias to isolate and label comments targeting LGBTQIA+ individuals along six distinct harmful language categories: Toxicity, Severe Toxicity, Obscene, Threat, Insult, and Identity Attack.
• Conducted human evaluation of annotators to ensure data quality and reliability in labeling sensitive, stereotypical, and offensive content.
• Implemented two baseline machine learning models—Support Vector Machine (SVM) and Logistic Regression—for binary and multi-label classification of harmful content.
• Fine-tuned three pre-trained large language models—BERT, RoBERTa, and HateBERT—on the dataset to assess performance in detecting anti-LGBTQIA+ sentiment and language.
• Used automated labeling via Detoxify to validate and cross-check human-annotated labels, ensuring consistency across the dataset.
• Released the full multi-label dataset and code publicly to support future research and shared tasks in anti-LGBTQIA+ content detection.

Experimental results

Research questions

• RQ1 Can a multi-label dataset effectively capture the nuanced forms of anti-LGBTQIA+ harmful language, including identity attacks, threats, and offensive stereotypes?
• RQ2 How do traditional machine learning models compare to fine-tuned large language models in detecting anti-LGBTQIA+ conversational toxicity?
• RQ3 To what extent do pre-trained language models like BERT and HateBERT generalize to detecting subtle or context-dependent anti-LGBTQIA+ slurs and microaggressions?
• RQ4 What are the performance trade-offs when detecting low-resource labels such as 'Threat' due to class imbalance in the dataset?

Key findings

• BERT achieved an F1-score of at least 0.82 across all but the 'Threat' label, demonstrating strong performance in detecting anti-LGBTQIA+ harmful content.
• HateBERT, despite being pre-trained for hate speech detection, underperformed compared to BERT, indicating that general-purpose LLMs may be more effective for this task.
• The dataset revealed that identity attacks and insults were the most frequent harmful categories, with high toxicity scores across examples from Reddit.
• The 'Threat' label had the lowest number of samples, resulting in the lowest F1-score across all models, highlighting data imbalance as a key challenge.
• Human-annotated labels showed high consistency, validating the reliability of the dataset for future shared tasks and model evaluation.
• The study confirms that large language models can be effectively fine-tuned to detect anti-LGBTQIA+ conversational content with high precision and recall.