[Paper Review] ToKen: Task Decomposition and Knowledge Infusion for Few-Shot Hate Speech Detection

Badr AlKhamissi, Faisal Ladhak+8 more|arXiv (Cornell University)|May 25, 2022

Hate Speech and Cyberbullying Detection4 citations

TL;DR

This paper proposes TOKEN, a few-shot hate speech detection method that decomposes the task into subtasks (offensiveness, target group detection) and infuses commonsense knowledge via fine-tuning on ATOMIC²⁰ and StereoSet. It achieves a 17.83% absolute F1 improvement over baseline models in the 16-shot setting, with enhanced generalization and robustness across out-of-distribution datasets and training variations.

ABSTRACT

Hate speech detection is complex; it relies on commonsense reasoning, knowledge of stereotypes, and an understanding of social nuance that differs from one culture to the next. It is also difficult to collect a large-scale hate speech annotated dataset. In this work, we frame this problem as a few-shot learning task, and show significant gains with decomposing the task into its "constituent" parts. In addition, we see that infusing knowledge from reasoning datasets (e.g. Atomic2020) improves the performance even further. Moreover, we observe that the trained models generalize to out-of-distribution datasets, showing the superiority of task decomposition and knowledge infusion compared to previously used methods. Concretely, our method outperforms the baseline by 17.83% absolute gain in the 16-shot case.

Motivation & Objective

• To address the challenge of few-shot hate speech detection, where labeled data is scarce and models often fail to generalize.
• To improve model performance and robustness by decomposing hate speech detection into interpretable subtasks: offensiveness and group targeting.
• To enhance model reasoning capabilities by infusing commonsense and stereotype knowledge through pre-fine-tuning on ATOMIC²⁰ and StereoSet.
• To evaluate generalization to out-of-distribution datasets and robustness to data partitioning and hyperparameter variation.
• To enable more explainable systems by isolating failure modes in subtasks and improving them individually.

Proposed method

• The model decomposes hate speech detection into two subtasks: classifying whether a post is offensive and identifying if a group is targeted.
• The BART-Large model is pre-fine-tuned on the ATOMIC²⁰ dataset to inject commonsense reasoning and on StereoSet to improve knowledge of social stereotypes.
• The final model performs end-to-end training on the SBIC dataset using a conditional generation framework with subtask-specific heads.
• Performance is evaluated across 10 random seeds, with stratified sampling of few-shot training sets from 16 to 1024 examples.
• Generalization is tested on three out-of-distribution hate speech datasets, and robustness is measured via variance across seeds, data splits, and hyperparameters.
• The method enables model interpretability by isolating subtask-level performance, allowing targeted improvements.

Experimental results

Research questions

• RQ1 Can task decomposition into offensiveness and group-targeting subtasks improve few-shot hate speech detection performance?
• RQ2 Does pre-fine-tuning on reasoning and stereotype knowledge datasets (ATOMIC²⁰ and StereoSet) enhance model performance in few-shot settings?
• RQ3 Do models trained with task decomposition generalize better to out-of-distribution hate speech datasets compared to end-to-end binary classification baselines?
• RQ4 How does the variance in model performance change across different random seeds, data splits, and hyperparameters when using task decomposition and knowledge infusion?
• RQ5 Can subtask-level analysis identify failure modes and guide targeted improvements in overall hate speech detection?

Key findings

• TOKEN achieves a 17.83% absolute F1 improvement over the baseline in the 16-shot setting, demonstrating significant performance gains from task decomposition and knowledge infusion.
• The model generalizes better to three out-of-distribution hate speech datasets, indicating improved robustness and transferability.
• Performance variance across 10 seeds, data partitions, and hyperparameters is significantly lower for TOKEN models, showing enhanced robustness.
• Subtask-level analysis reveals that targeting specific bottlenecks—such as group detection—leads to measurable improvements in overall model performance.
• Fine-tuning on ATOMIC²⁰ and StereoSet improves reasoning and stereotype understanding, which directly contributes to better hate speech detection.
• The method enables more explainable systems by allowing failure analysis at the subtask level, moving toward interpretable and fixable model behavior.