[论文解读] NoisyHate: Mining Online Human-Written Perturbations for Realistic Robustness Benchmarking of Content Moderation Models

Yiran Ye, Thái Hoàng Lê+1 more|arXiv (Cornell University)|Mar 18, 2023

Hate Speech and Cyberbullying Detection被引用 4

一句话总结

本文提出了 NoisyHate，一个来自社交媒体的真实人类书写的文本扰动基准数据集，旨在评估毒性言论检测模型在真实场景下的对抗性规避攻击中的鲁棒性。结果表明，即使是最先进的模型（如 BERT 和 RoBERTa）以及商业化的 Perspective API，仍无法可靠检测出经过扰动的毒性内容，凸显了当前模型在鲁棒性方面存在关键性缺陷，尽管它们对扰动的理解已接近人类水平。

ABSTRACT

Online texts with toxic content are a clear threat to the users on social media in particular and society in general. Although many platforms have adopted various measures (e.g., machine learning-based hate-speech detection systems) to diminish their effect, toxic content writers have also attempted to evade such measures by using cleverly modified toxic words, so-called human-written text perturbations. Therefore, to help build automatic detection tools to recognize those perturbations, prior methods have developed sophisticated techniques to generate diverse adversarial samples. However, we note that these ``algorithms"-generated perturbations do not necessarily capture all the traits of ``human"-written perturbations. Therefore, in this paper, we introduce a novel, high-quality dataset of human-written perturbations, named as NoisyHate, that was created from real-life perturbations that are both written and verified by human-in-the-loop. We show that perturbations in NoisyHate have different characteristics than prior algorithm-generated toxic datasets show, and thus can be in particular useful to help develop better toxic speech detection solutions. We thoroughly validate NoisyHate against state-of-the-art language models, such as BERT and RoBERTa, and black box APIs, such as Perspective API, on two tasks, such as perturbation normalization and understanding.

研究动机与目标

• 解决毒性内容检测中机器生成的对抗性扰动与真实人类书写的扰动之间的差距。
• 通过使用来自在线社交平台的真实扰动，创建一个真实的基准数据集，以提升模型评估的可靠性。
• 评估现有模型和拼写检查器在人类书写的扰动上的有效性。
• 证明当前模型对现实世界中人类生成的对抗性攻击依然脆弱。

提出的方法

• 通过在 Amazon MTurk 上进行众包，从社交媒体平台收集真实的人类书写的扰动，并使用精心设计的指南确保其真实性。
• 以 Jigsaw 毒性评论数据集为基准，保留原始的毒性标签和身份标注。
• 应用多种拼写检查器（例如，pyspellchecker、Google 搜索）以评估归一化的可行性并识别其局限性。
• 在数据集的干净版本和扰动版本上，评估最先进模型（BERT、RoBERTa）以及 Perspective API 的表现。
• 通过在不同决策阈值下进行基于阈值的准确率比较，衡量模型在扰动下的鲁棒性。
• 设计了一个交互式的人工评估界面，以验证扰动的质量和人类相似性。

Figure 1 . Overall curation pipeline of NoisyHate dataset. This pipeline has three steps: (1) Data sourcing and cleaning from the original Jigsaw dataset (Section 3.2 ), (2) Sentence perturbation with human-written perturbations via pseudo-random sampling (Section 3.3 ) and (3) Human evaluation via

实验结果

研究问题

• RQ1 当前的毒性内容检测模型在识别真实人类书写的文本扰动方面效果如何？
• RQ2 现有的拼写检查器能否有效归一化在线毒性内容中发现的人类书写扰动？
• RQ3 当面对真实人类生成的对抗性扰动时，最先进模型的性能如何下降？
• RQ4 哪些类型的扰动在逃避检测系统方面最为普遍且有效？

主要发现

• Perspective API 在扰动数据集上表现出最高的鲁棒性，阈值为 0.5 时准确率为 0.672，优于 BERT（0.557）和 RoBERTa（0.583）。
• RoBERTa-TweetEval 在干净数据上表现最佳（阈值 t=0.5 时准确率为 0.915），但在扰动数据上的性能显著下降。
• 像 pyspellchecker 这类拼写检查器在归一化方面表现优于商业工具，但仍无法处理新颖或复杂的扰动。
• 该数据集表明，即使先进的模型在面对人类书写的扰动时也难以维持高检测准确率，凸显了关键的鲁棒性缺陷。
• 成功收集并验证了如 'russ!an' 和 'idiiot' 等人类书写的扰动，作为毒性词汇的现实、非机械的变体。

Figure 2 . Python code for loading the NoisyHate datasets into a table using the Hugging Face API