[Paper Review] L3Cube-MahaHate: A Tweet-based Marathi Hate Speech Detection Dataset and BERT models

Abhishek Velankar, Hrushikesh Patil+3 more|arXiv (Cornell University)|Mar 25, 2022

Hate Speech and Cyberbullying DetectionComputer Science25 citations

TL;DR

The paper introduces L3Cube-MahaHate, a 25k-tweet Marathi hate speech dataset with four labels, and benchmarks CNN, LSTM, BiLSTM, and transformers, showing monolingual Marathi BERTs outperform multilingual ones; MahaBERT achieves best binary accuracy and MahaRoBERTa best four-class accuracy.

ABSTRACT

Social media platforms are used by a large number of people prominently to express their thoughts and opinions. However, these platforms have contributed to a substantial amount of hateful and abusive content as well. Therefore, it is important to curb the spread of hate speech on these platforms. In India, Marathi is one of the most popular languages used by a wide audience. In this work, we present L3Cube-MahaHate, the first major Hate Speech Dataset in Marathi. The dataset is curated from Twitter, annotated manually. Our dataset consists of over 25000 distinct tweets labeled into four major classes i.e hate, offensive, profane, and not. We present the approaches used for collecting and annotating the data and the challenges faced during the process. Finally, we present baseline classification results using deep learning models based on CNN, LSTM, and Transformers. We explore mono-lingual and multi-lingual variants of BERT like MahaBERT, IndicBERT, mBERT, and xlm-RoBERTa and show that mono-lingual models perform better than their multi-lingual counterparts. The MahaBERT model provides the best results on L3Cube-MahaHate Corpus. The data and models are available at https://github.com/l3cube-pune/MarathiNLP .

Motivation & Objective

• Create a large, manually annotated Marathi hate speech dataset from Twitter with fine-grained labels (Hate, Offensive, Profane, Not).
• Provide baseline benchmarks using CNN, LSTM, BiLSTM, and transformer models including monolingual and multilingual BERT variants.
• Compare mono-lingual Marathi BERT models with multilingual counterparts to assess performance advantages.
• Offer publicly available datasets and models for Marathi NLP and hate speech detection.

Proposed method

• Data collection from Twitter using Marathi hate-related keywords and over 150 bad words; manual annotation by native Marathi speakers into four classes.
• Dataset statistics with train/val/test splits and uniform class distribution to simplify modeling.
• Baseline experiments with CNN, LSTM, and BiLSTM using random, trainable, and non-trainable fastText embeddings.
• Transformer-based models including IndicBERT, mBERT, xlm-RoBERTa, MahaBERT, MahaALBERT, MahaRoBERTa; fine-tuning and evaluation on 2-class and 4-class tasks.
• Public release of MahaHate-BERT and MahaHate-MultiRoberta variants on model hubs; resources on GitHub.

Experimental results

Research questions

• RQ1 How well do state-of-the-art deep learning architectures perform on the Marathi hate speech dataset for binary and four-class classification?
• RQ2 Do monolingual Marathi BERT models outperform multilingual BERT variants for Marathi hate speech detection?
• RQ3 Which transformer variants yield the best accuracy for 2-class and 4-class Marathi hate speech classification?
• RQ4 What is the impact of using trainable vs non-trainable embeddings on CNN/LSTM baselines?

Key findings

• CNN/LSTM/BiLSTM baselines with non-trainable fastText embeddings yield competitive performance, often surpassing trainable variants.
• Monolingual Marathi BERT models outperform multilingual counterparts across tasks.
• MahaBERT achieves the best 2-class (binary) accuracy of 0.909, while MahaRoBERTa achieves the best 4-class accuracy of 0.803.
• Among transformer variants, MahaBERT and MahaRoBERTa deliver top results in binary and 4-class setups respectively.
• The best reported results are 2-class: 0.909 (MahaBERT); 4-class: 0.803 (MahaRoBERTa).