[Paper Review] NLP-CUET@LT-EDI-EACL2021: Multilingual Code-Mixed Hope Speech Detection using Cross-lingual Representation Learner

Eftekhar Hossain, Omar Sharif+1 more|arXiv (Cornell University)|Jan 1, 2021

Hate Speech and Cyberbullying Detection19 references7 citations

TL;DR

This paper proposes a multilingual code-mixed hope speech detection system using cross-lingual transformer models, with XLM-RoBERTa achieving state-of-the-art performance across English, Tamil, and Malayalam, attaining weighted F1-scores of 0.931, 0.602, and 0.854 respectively. The approach leverages fine-tuned pre-trained transformers on a multilingual, code-mixed dataset from social media, outperforming traditional ML and DL methods.

ABSTRACT

In recent years, several systems have been developed to regulate the spread of negativity and eliminate aggressive, offensive or abusive contents from the online platforms. Nevertheless, a limited number of researches carried out to identify positive, encouraging and supportive contents. In this work, our goal is to identify whether a social media post/comment contains hope speech or not. We propose three distinct models to identify hope speech in English, Tamil and Malayalam language to serve this purpose. To attain this goal, we employed various machine learning (support vector machine, logistic regression, ensemble), deep learning (convolutional neural network + long short term memory) and transformer (m-BERT, Indic-BERT, XLNet, XLM-Roberta) based methods. Results indicate that XLM-Roberta outdoes all other techniques by gaining a weighted $f_1$-score of $0.93$, $0.60$ and $0.85$ respectively for English, Tamil and Malayalam language. Our team has achieved $1^{st}$, $2^{nd}$ and $1^{st}$ rank in these three tasks respectively.

Motivation & Objective

• To develop a computational model that detects hope speech—positive, supportive, and encouraging content—in multilingual, code-mixed social media posts.
• To address the scarcity of annotated datasets and the challenges of multilingual and code-mixed text in hope speech detection.
• To evaluate and compare the performance of diverse models, including traditional machine learning, deep learning, and transformer-based architectures.
• To achieve high accuracy in classifying hope speech, not hope speech, and not intended language (NIL) across English, Tamil, and Malayalam.

Proposed method

• Fine-tuned XLM-RoBERTa, m-BERT, Indic-BERT, XLNet, and BERT-based models on a multilingual, code-mixed hope speech dataset for English, Tamil, and Malayalam.
• Employed transfer learning via cross-lingual representation learners to capture semantic and syntactic patterns across languages.
• Used TF-IDF and FastText embeddings as baseline features for traditional machine learning and deep learning models.
• Combined CNN and BiLSTM architectures with Keras and FastText embeddings to model sequential and local patterns in text.
• Applied early stopping and the Ktrain 'fit onecycle' method for efficient fine-tuning of transformer models over 30 epochs with a 2e−5 initial learning rate.
• Conducted extensive hyperparameter tuning using validation sets and evaluated final performance on unseen test sets.

Experimental results

Research questions

• RQ1 How do traditional machine learning and deep learning models compare to state-of-the-art transformer models in detecting code-mixed hope speech across multiple languages?
• RQ2 What is the impact of multilingual pre-training and cross-lingual transfer on hope speech detection performance?
• RQ3 Why does XLM-RoBERTa outperform other transformer models in this multilingual, code-mixed setting?
• RQ4 How do class imbalance and code-mixing affect model generalization and misclassification patterns?
• RQ5 Can multilingual models effectively detect hope speech in low-resource languages like Tamil and Malayalam?

Key findings

• XLM-RoBERTa achieved the highest weighted F1-score of 0.931 on the English test set, outperforming all other models.
• For Tamil, XLM-RoBERTa achieved a weighted F1-score of 0.602, surpassing m-BERT (0.588), Indic-BERT (0.578), and XLNet (0.558).
• In Malayalam, XLM-RoBERTa achieved a weighted F1-score of 0.854, exceeding Indic-BERT (0.840) and m-BERT (0.804).
• The ensemble model achieved the best performance among traditional ML models with a weighted F1-score of 0.905 on English and 0.573 on Tamil.
• The confusion matrix revealed that the model most frequently confused hope speech (HS) with not hope speech (NHS), especially due to code-mixing and class imbalance.
• Despite high performance, the model struggled with the 'not intended language' (NIL) class, misclassifying many low-resource, short texts as NHS due to limited training examples.