[Paper Review] Code-mixed Sentiment and Hate-speech Prediction

Anjali Yadav, Tanya Garg+4 more|arXiv (Cornell University)|May 21, 2024

Hate Speech and Cyberbullying Detection4 citations

TL;DR

This paper introduces four new bilingual masked language models for English-Hindi and English-Slovene code-mixed text, specifically pre-trained on informal social media data. It evaluates monolingual, bilingual, multilingual, and generative models on sentiment analysis and hate-speech detection, finding that fine-tuned bilingual and social-media-specialized multilingual models outperform general-purpose and large generative models, with HingRoBERTa achieving the highest F1 scores in Hindi and SlEng-BERT in Slovene.

ABSTRACT

Code-mixed discourse combines multiple languages in a single text. It is commonly used in informal discourse in countries with several official languages, but also in many other countries in combination with English or neighboring languages. As recently large language models have dominated most natural language processing tasks, we investigated their performance in code-mixed settings for relevant tasks. We first created four new bilingual pre-trained masked language models for English-Hindi and English-Slovene languages, specifically aimed to support informal language. Then we performed an evaluation of monolingual, bilingual, few-lingual, and massively multilingual models on several languages, using two tasks that frequently contain code-mixed text, in particular, sentiment analysis and offensive language detection in social media texts. The results show that the most successful classifiers are fine-tuned bilingual models and multilingual models, specialized for social media texts, followed by non-specialized massively multilingual and monolingual models, while huge generative models are not competitive. For our affective problems, the models mostly perform slightly better on code-mixed data compared to non-code-mixed data.

Motivation & Objective

• To address the lack of effective large language models for code-mixed sentiment and hate-speech analysis in multilingual, informal settings.
• To create new bilingual masked language models specifically trained on informal code-mixed corpora for English-Hindi and English-Slovene.
• To evaluate the performance of diverse LLM types—monolingual, bilingual, multilingual, and generative—on code-mixed affective computing tasks.
• To investigate whether code-mixed data is inherently easier to classify than non-code-mixed data for sentiment and offensive language detection.
• To assess the limitations of existing language detection tools in identifying code-mixed text.

Proposed method

• Pre-trained four new masked language models: RoBERTa-en-hi-codemixed and MuRIL-en-hi-codemixed for English-Hindi, and SlEng-BERT and SloBERTa-SlEng for English-Slovene, using code-mixed social media corpora.
• Fine-tuned these models on two affective computing tasks: sentiment analysis and offensive language detection across five languages (French, Hindi, Russian, Slovene, Tamil).
• Evaluated a range of LLMs, including mBERT, XLM-R, TwHIN-BERT, GPT-3, and Llama2-7B, on both code-mixed and non-code-mixed data subsets.
• Used standard metrics like F1-score to compare model performance across language pairs and data types.
• Conducted manual inspection and contrast set analysis to assess model behavior and language detection weaknesses.
• Applied transfer learning techniques to adapt pre-trained models to low-resource and code-mixed settings.

Experimental results

Research questions

• RQ1 How do bilingual and multilingual language models perform on code-mixed sentiment and hate-speech detection compared to monolingual and massive multilingual models?
• RQ2 Does training on informal, code-mixed social media data improve model performance on affective computing tasks?
• RQ3 Is code-mixed data inherently easier to classify than non-code-mixed data for sentiment and offensive language detection?
• RQ4 How do large generative models like GPT-3 and Llama2-7B perform on code-mixed affective tasks compared to specialized models?
• RQ5 What are the limitations of existing language detection tools in identifying code-mixed text?

Key findings

• The bilingual HingRoBERTa model achieved the highest F1 score of 0.876 on sentiment analysis for Hindi-English code-mixed data, outperforming all other models.
• The newly introduced RoBERTa-en-hi-codemixed and MuRIL-en-hi-codemixed models showed better performance on code-mixed data than on non-code-mixed data in Hindi.
• For Slovene-English code-mixing, SlEng-BERT achieved an F1 score of 0.874 on sentiment analysis and 0.865 on offensive language detection, outperforming other models.
• Models trained on social media-specific code-mixed corpora, such as HingRoBERTa and SlEng-BERT, consistently outperformed general-purpose multilingual and monolingual models.
• GPT-3 and Llama2-7B showed suboptimal performance, with GPT-3 achieving only 0.466 F1 on Hindi sentiment analysis, indicating limited adaptability to code-mixed text.
• Surprisingly, most models performed slightly better on code-mixed data than on non-code-mixed data, suggesting potential shared affective patterns across languages in code-mixed discourse.