[Paper Review] SOLID: A Large-Scale Semi-Supervised Dataset for Offensive Language Identification

Sara Brin Rosenthal, Pepa Atanasova+3 more|arXiv (Cornell University)|Apr 29, 2020

Hate Speech and Cyberbullying Detection4 citations

TL;DR

This paper introduces SOLID, a large-scale semi-supervised dataset of 9 million English tweets for offensive language identification, created by leveraging the OLID dataset as a seed and applying ensemble semi-supervised learning to expand coverage. The method achieves significant performance gains—especially on lower taxonomy levels—when fine-tuning models on SOLID and evaluating on OLID, demonstrating its effectiveness for low-resource offensive language detection tasks.

ABSTRACT

The widespread use of offensive content in social media has led to an abundance of research in detecting language such as hate speech, cyberbullying, and cyber-aggression. Recent work presented the OLID dataset, which follows a taxonomy for offensive language identification that provides meaningful information for understanding the type and the target of offensive messages. However, it is limited in size and it might be biased towards offensive language as it was collected using keywords. In this work, we present SOLID, an expanded dataset, where the tweets were collected in a more principled manner. SOLID contains over nine million English tweets labeled in a semi-supervised fashion. We demonstrate that using SOLID along with OLID yields sizable performance gains on the OLID test set for two different models, especially for the lower levels of the taxonomy.

Motivation & Objective

• To address the scarcity of labeled offensive language data, especially at lower levels of hierarchical taxonomy, which hinders robust model training.
• To develop a scalable, cost-effective method for collecting large-scale offensive language data without relying solely on expensive manual annotation.
• To create a publicly available, large-scale dataset that supports improved performance in offensive language identification, particularly for implicit or subtle offensive content.
• To mitigate biases in data collection by using frequent stop words as query terms and employing ensemble models with diverse inductive biases.
• To enable better generalization and robustness in offensive language detection systems through a larger, more representative training distribution.

Proposed method

• The authors use the OLID dataset as a seed for semi-supervised learning, applying a multi-stage pipeline to expand the training data.
• They collect tweets using frequent stop words as query terms to ensure broad, representative coverage of natural language usage.
• An ensemble of diverse pre-trained language models is trained on the OLID seed data and used to predict labels for millions of unlabeled tweets.
• The model predictions are filtered and aggregated using confidence thresholds and consensus voting to reduce noise and bias.
• The final dataset, SOLID, contains 9 million English tweets with hierarchical labels across three levels: offensive detection, type classification, and target identification.
• The authors validate the quality of the semi-supervised labels by comparing model performance on OLID test set when fine-tuned on SOLID versus OLID alone.

Experimental results

Research questions

• RQ1 Can semi-supervised learning significantly improve offensive language detection performance when training data is limited?
• RQ2 Does expanding the training data using a seed dataset and model ensembles lead to better generalization, especially on low-resource taxonomy levels?
• RQ3 How does the quality of semi-supervised labels compare to gold-standard annotations in terms of model performance and bias mitigation?
• RQ4 Can a large-scale dataset like SOLID improve detection of implicit offensive content (e.g., sarcasm, coded slurs) compared to smaller, manually curated datasets?
• RQ5 To what extent does the choice of initial query terms (e.g., stop words vs. keywords) affect the representativeness and bias of the resulting dataset?

Key findings

• Fine-tuning models on SOLID leads to substantial performance gains on the OLID test set, particularly on the lower levels of the taxonomy, which are under-resourced in existing datasets.
• The performance improvement is most pronounced for the target identification level (Level C), where the number of training instances is smallest, indicating the effectiveness of the semi-supervised expansion.
• The ensemble semi-supervised approach reduces label noise and biases compared to single-model predictions, as evidenced by higher inter-annotator agreement and robustness across diverse model architectures.
• The dataset includes a comprehensive analysis of EASY (explicit) and HARD (implicit) offensive examples, showing that models trained on SOLID generalize better to subtle or indirect forms of offense.
• The authors demonstrate that the ratio of offensive to non-offensive tweets in SOLID is more balanced and representative than in keyword-based datasets, reducing selection bias.
• The dataset is publicly released and has been adopted as the official training set for the SemEval-2020 OffensEval shared task, confirming its utility and reliability in the research community.