[Paper Review] Cross-lingual Zero- and Few-shot Hate Speech Detection Utilising Frozen Transformer Language Models and AXEL

Lukas Stappen, Fabian Brunn+1 more|arXiv (Cornell University)|Apr 13, 2020

Hate Speech and Cyberbullying Detection49 references52 citations

TL;DR

The paper proposes a frozen-transformer feature-extraction approach with a novel AXEL attention-based classification block to tackle uni-, cross-lingual zero-/few-shot hate speech detection on HatEval English and Spanish data, achieving competitive results with far fewer trainable parameters.

ABSTRACT

Detecting hate speech, especially in low-resource languages, is a non-trivial challenge. To tackle this, we developed a tailored architecture based on frozen, pre-trained Transformers to examine cross-lingual zero-shot and few-shot learning, in addition to uni-lingual learning, on the HatEval challenge data set. With our novel attention-based classification block AXEL, we demonstrate highly competitive results on the English and Spanish subsets. We also re-sample the English subset, enabling additional, meaningful comparisons in the future.

Motivation & Objective

• Motivate cross-lingual hate speech detection in low-resource languages using frozen transformer features.
• Evaluate uni-lingual and cross-lingual (zero-/few-shot) learning on English and Spanish HatEval data.
• Develop and assess a lightweight classification block (AXEL) to maximize information from frozen transformer features.
• Propose a stratified English data partition to mitigate out-of-domain sampling and better assess generalization.

Proposed method

• Use frozen Transformer Language Models (BERT base multilingual and XLM) as feature extractors rather than fine-tuning.
• Extract representations from selected layers and feed them into trainable classification blocks.
• Introduce AXEL, an attention-based block inspired by vision modules to compress and enhance sequential text features before classification.
• Compare multiple classification blocks (including RCAB, CBAM, CSAR, RAM) and demonstrate AXEL’s superior performance.
• Evaluate zero-shot and few-shot cross-lingual transfer by training on one language and testing on another, and by injecting small portions of target-language data.

Experimental results

Research questions

• RQ1 How effective are frozen Transformer features for uni-lingual hate speech detection compared to fine-tuned models?
• RQ2 Can AXEL improve classification performance when using frozen Transformer representations for hate speech detection?
• RQ3 What is the impact of cross-lingual zero-shot and few-shot learning on English and Spanish HatEval data?
• RQ4 Does stratified English data partitioning reduce out-of-domain sampling effects and improve generalization?
• RQ5 How do cross-lingual representations (XLM-based) compare to BERT-based features for cross-language hate speech detection?

Key findings

• Frozen Transformer features with AXEL achieve competitive hate speech detection results with far fewer trainable parameters than fine-tuned models.
• AXEL substantially outperforms other adapted blocks, with EN-S achieving 71.16 F1 and ES achieving 69.70 F1 in Table 4 results.
• Cross-lingual zero-shot performance is generally weaker than monolingual, but AXEL remains the best among XLM-based options on most settings; zero-shot can improve with translation-augmented evaluation.
• Few-shot learning shows substantial gains; adding as little as 1% target-language data dramatically boosts F1 and can even surpass monolingual performance in some setups (e.g., EN-S).
• A new English data partition (EN-S) reduces out-of-domain sampling effects and yields more balanced, comparable performance across EN and ES subsets.
• XLM-based models are strong in zero-shot settings when using a simple dense/AXEL classifier, while BERT-based sequential encoders benefit more from full-sequence encoding.