[Paper Review] Overview of Abusive and Threatening Language Detection in Urdu at FIRE 2021

Maaz Amjad, Alisa Zhila+6 more|arXiv (Cornell University)|Jul 14, 2022

Hate Speech and Cyberbullying Detection4 citations

TL;DR

This paper presents a shared task for abusive and threatening language detection in Urdu, introducing two manually annotated datasets—2,400 training and 1,100 test tweets for abuse detection, and 6,000 training and 3,950 test tweets for threat detection. The best-performing system, using a multilingual BERT (m-BERT) model, achieved an F1 score of 0.880 on the abusive language detection task and 0.545 on the threat detection task, highlighting the effectiveness of transformer-based models for low-resource Urdu NLP tasks.

ABSTRACT

With the growth of social media platform influence, the effect of their misuse becomes more and more impactful. The importance of automatic detection of threatening and abusive language can not be overestimated. However, most of the existing studies and state-of-the-art methods focus on English as the target language, with limited work on low- and medium-resource languages. In this paper, we present two shared tasks of abusive and threatening language detection for the Urdu language which has more than 170 million speakers worldwide. Both are posed as binary classification tasks where participating systems are required to classify tweets in Urdu into two classes, namely: (i) Abusive and Non-Abusive for the first task, and (ii) Threatening and Non-Threatening for the second. We present two manually annotated datasets containing tweets labelled as (i) Abusive and Non-Abusive, and (ii) Threatening and Non-Threatening. The abusive dataset contains 2400 annotated tweets in the train part and 1100 annotated tweets in the test part. The threatening dataset contains 6000 annotated tweets in the train part and 3950 annotated tweets in the test part. We also provide logistic regression and BERT-based baseline classifiers for both tasks. In this shared task, 21 teams from six countries registered for participation (India, Pakistan, China, Malaysia, United Arab Emirates, and Taiwan), 10 teams submitted their runs for Subtask A, which is Abusive Language Detection and 9 teams submitted their runs for Subtask B, which is Threatening Language detection, and seven teams submitted their technical reports. The best performing system achieved an F1-score value of 0.880 for Subtask A and 0.545 for Subtask B. For both subtasks, m-Bert based transformer model showed the best performance.

Motivation & Objective

• To address the lack of research on abusive and threatening language detection in low-resource languages, particularly Urdu, which has over 170 million speakers.
• To promote the development of robust NLP systems for Urdu by organizing a shared task within the FIRE 2021 and ODS SoC 2021 frameworks.
• To provide publicly available, high-quality, manually annotated datasets for abusive and threatening language in Urdu to support future research.
• To evaluate diverse NLP approaches—ranging from traditional machine learning to state-of-the-art transformer models—on Urdu text classification tasks.
• To foster international collaboration by engaging 21 teams from six countries in solving Urdu-specific NLP challenges related to online safety.

Proposed method

• Two binary classification shared tasks were conducted: Subtask A for abusive vs. non-abusive language and Subtask B for threatening vs. non-threatening content in Urdu tweets.
• The datasets were constructed using Twitter data, with annotations following Twitter’s official definitions of abusive and threatening behavior.
• Participants were provided with training sets containing ground-truth labels and test sets with hidden labels for evaluation.
• Baseline models were implemented using logistic regression and multilingual BERT (m-BERT), with additional baselines including XLM-RoBERTa for comparison.
• Evaluation was performed using F1-score and ROC-AUC metrics, with results ranked primarily by F1-score on the private test set.
• Late submissions were excluded from official ranking due to potential data leakage after ground-truth release, ensuring fairness in evaluation.

Experimental results

Research questions

• RQ1 How effective are multilingual transformer models such as m-BERT and XLM-RoBERTa in detecting abusive and threatening language in low-resource Urdu text?
• RQ2 What performance gap exists between traditional machine learning approaches and deep learning-based methods in Urdu abusive and threat detection?
• RQ3 To what extent do feature engineering and pre-trained embeddings contribute to classification performance compared to end-to-end contextual modeling?
• RQ4 How does the performance of systems vary across different linguistic and stylistic patterns in Urdu, especially in code-mixed or informal contexts?
• RQ5 What are the challenges and limitations of existing datasets in supporting generalization and robustness in real-world abusive content detection?

Key findings

• The winning system for Subtask A (abusive language detection) achieved an F1 score of 0.880 using the m-BERT model, significantly outperforming traditional machine learning baselines.
• For Subtask B (threatening language detection), the best official result was an F1 score of 0.545, also achieved by the m-BERT model, indicating higher difficulty in detecting threats compared to abuse.
• A late submission from team SSNCSE_NLP achieved a much higher F1 score of 0.805 on Subtask B, suggesting potential data leakage after the official competition closed.
• The m-BERT model consistently outperformed other approaches across both tasks, demonstrating strong transferability and robustness for Urdu NLP tasks.
• Traditional models such as SVM, logistic regression, and random forests showed limited performance, with F1 scores below 0.60 on both tasks, indicating the need for contextual representations.
• The public release of the annotated datasets, including training and test splits, enables future research and reproducibility, with the potential to improve performance through larger-scale data and model fine-tuning.