[Paper Review] NoisyHate: Mining Online Human-Written Perturbations for Realistic Robustness Benchmarking of Content Moderation Models

Yiran Ye, Thái Hoàng Lê+1 more|arXiv (Cornell University)|Mar 18, 2023

Hate Speech and Cyberbullying Detection4 citations

TL;DR

This paper introduces NoisyHate, a benchmark dataset of real human-written text perturbations from social media, designed to evaluate the robustness of toxic speech detection models against realistic evasion attacks. It demonstrates that state-of-the-art models like BERT and RoBERTa, as well as the commercial Perspective API, still fail to reliably detect perturbed toxic content, highlighting a critical gap in current model robustness despite human-like understanding of the perturbations.

ABSTRACT

Online texts with toxic content are a clear threat to the users on social media in particular and society in general. Although many platforms have adopted various measures (e.g., machine learning-based hate-speech detection systems) to diminish their effect, toxic content writers have also attempted to evade such measures by using cleverly modified toxic words, so-called human-written text perturbations. Therefore, to help build automatic detection tools to recognize those perturbations, prior methods have developed sophisticated techniques to generate diverse adversarial samples. However, we note that these ``algorithms"-generated perturbations do not necessarily capture all the traits of ``human"-written perturbations. Therefore, in this paper, we introduce a novel, high-quality dataset of human-written perturbations, named as NoisyHate, that was created from real-life perturbations that are both written and verified by human-in-the-loop. We show that perturbations in NoisyHate have different characteristics than prior algorithm-generated toxic datasets show, and thus can be in particular useful to help develop better toxic speech detection solutions. We thoroughly validate NoisyHate against state-of-the-art language models, such as BERT and RoBERTa, and black box APIs, such as Perspective API, on two tasks, such as perturbation normalization and understanding.

Motivation & Objective

• To address the gap between machine-generated adversarial perturbations and real human-written perturbations in toxic content detection.
• To create a realistic benchmark dataset using actual perturbations from online social platforms to improve model evaluation.
• To evaluate the effectiveness of existing models and spell checkers on human-written perturbations.
• To demonstrate that current models remain vulnerable to real-world human-generated adversarial attacks.

Proposed method

• Collected real human-written perturbations from social media platforms through crowd-sourcing on Amazon MTurk, using curated guidelines to ensure authenticity.
• Anchored the dataset on the Jigsaw toxic comment dataset, preserving original toxicity labels and identity annotations.
• Applied multiple spell checkers (e.g., pyspellchecker, Google Search) to assess normalization feasibility and identify limitations.
• Evaluated state-of-the-art models (BERT, RoBERTa) and the Perspective API on both clean and perturbed versions of the dataset.
• Used a threshold-based accuracy comparison across varying decision thresholds to measure robustness under perturbations.
• Designed an interactive human evaluation interface to validate the quality and human-likeness of perturbations.

Figure 1 . Overall curation pipeline of NoisyHate dataset. This pipeline has three steps: (1) Data sourcing and cleaning from the original Jigsaw dataset (Section 3.2 ), (2) Sentence perturbation with human-written perturbations via pseudo-random sampling (Section 3.3 ) and (3) Human evaluation via

Experimental results

Research questions

• RQ1 How effective are current toxic content detection models in recognizing real human-written text perturbations?
• RQ2 Can existing spell checkers effectively normalize human-written perturbations found in online toxic content?
• RQ3 How does the performance of state-of-the-art models degrade when faced with real human-generated adversarial perturbations?
• RQ4 What types of perturbations are most prevalent and effective in evading detection systems?

Key findings

• The Perspective API achieved the highest robustness on the perturbed dataset, with an accuracy of 0.672 at a threshold of 0.5, outperforming BERT (0.557) and RoBERTa (0.583).
• RoBERTa-TweetEval achieved the best accuracy on clean data (0.915 at t=0.5), but its performance dropped significantly on perturbed data.
• Spell checkers like pyspellchecker showed higher accuracy in normalization than commercial tools, but still failed on novel or complex perturbations.
• The dataset revealed that even advanced models struggle to maintain high detection accuracy under human-written perturbations, indicating a critical robustness gap.
• Human-written perturbations such as 'russ!an' and 'idiiot' were successfully collected and validated as realistic, non-mechanical variants of toxic words.

Figure 2 . Python code for loading the NoisyHate datasets into a table using the Hugging Face API