[Paper Review] indicnlp@kgp at DravidianLangTech-EACL2021: Offensive Language Identification in Dravidian Languages

Kushal Kedia, Abhilash Nandy|arXiv (Cornell University)|Feb 14, 2021

Hate Speech and Cyberbullying Detection19 references9 citations

TL;DR

This paper presents a state-of-the-art ensemble system for offensive language identification in code-mixed Dravidian languages (Tamil-English, Malayalam-English, Kannada-English), leveraging transfer learning with multilingual BERT and RoBERTa, fine-tuned on additional monolingual English offensive language data (OLID). The approach achieved first, second, and third place with weighted F1 scores of 0.77, 0.97, and 0.72 on the respective datasets.

ABSTRACT

The paper presents the submission of the team indicnlp@kgp to the EACL 2021 shared task "Offensive Language Identification in Dravidian Languages." The task aimed to classify different offensive content types in 3 code-mixed Dravidian language datasets. The work leverages existing state of the art approaches in text classification by incorporating additional data and transfer learning on pre-trained models. Our final submission is an ensemble of an AWD-LSTM based model along with 2 different transformer model architectures based on BERT and RoBERTa. We achieved weighted-average F1 scores of 0.97, 0.77, and 0.72 in the Malayalam-English, Tamil-English, and Kannada-English datasets ranking 1st, 2nd, and 3rd on the respective tasks.

Motivation & Objective

• To address the challenge of offensive language identification in low-resource, code-mixed Dravidian language datasets.
• To improve model generalization in low-resource settings by leveraging transfer learning and external monolingual data.
• To investigate the effectiveness of ensemble models combining RNNs and transformer architectures for multilingual offensive language detection.
• To evaluate the impact of data augmentation via transfer learning on multilingual BERT and RoBERTa for code-mixed text.
• To rank competitively in the EACL 2021 shared task on offensive language identification in Dravidian languages.

Proposed method

• Fine-tuned multilingual BERT (mBERT) and XLM-RoBERTa (XLM-R) models on code-mixed Dravidian datasets using transfer learning.
• Combined the Dravidian code-mixed datasets with the monolingual OLID dataset (14k English tweets) to improve generalization and mitigate class imbalance.
• Applied data preprocessing including stopword removal, lemmatization, emoji replacement with descriptions, and mutual information-based feature selection.
• Trained an AWD-LSTM model with transfer learning via ULMFiT, using language model fine-tuning on domain-specific data.
• Constructed sentence representations via concatenation of max-pooling and mean-pooling of last-layer hidden states from transformer models.
• Built an ensemble model by averaging predictions from the top-performing XLM-R, mBERT, and ULMFiT models for final inference.

Experimental results

Research questions

• RQ1 Can transfer learning from monolingual English offensive language data (OLID) improve performance on low-resource code-mixed Dravidian language datasets?
• RQ2 How effective are multilingual transformer models (mBERT, XLM-R) when fine-tuned on combined Dravidian code-mixed data with external data?
• RQ3 To what extent does ensemble learning across diverse architectures (RNN, BERT, RoBERTa) improve robustness and performance in offensive language detection?
• RQ4 Does fine-tuning on a larger, combined dataset of Dravidian code-mixed texts enhance zero-shot or few-shot transfer to low-resource languages?
• RQ5 How do different pre-processing strategies (e.g., emoji replacement, lemmatization) impact model performance on code-mixed social media text?

Key findings

• The ensemble model achieved a weighted F1 score of 0.77 on the Tamil-English dataset, ranking 1st in the shared task.
• The model achieved a weighted F1 score of 0.97 on the Malayalam-English dataset, securing 2nd place.
• The model achieved a weighted F1 score of 0.72 on the Kannada-English dataset, ranking 3rd overall.
• Transfer learning using the OLID dataset significantly improved performance, especially on the smaller Kannada-English dataset.
• The XLM-RoBERTa model fine-tuned with combined data achieved the highest F1 scores across all three datasets, outperforming mBERT and ULMFiT.
• The ensemble of XLM-R, mBERT, and ULMFiT models yielded the most robust and highest-performing system, demonstrating the benefit of model diversity in low-resource settings.