[논문 리뷰] L3Cube-MahaHate: A Tweet-based Marathi Hate Speech Detection Dataset and BERT models

Abhishek Velankar, Hrushikesh Patil외 3명|arXiv (Cornell University)|2022. 03. 25.

Hate Speech and Cyberbullying Detection인용 수 25

한 줄 요약

본 논문은 L3Cube-MahaHate를 소개하고, 4개의 라벨이 있는 25k개의 마라티어 혐오발언 데이터셋을 제시하며 CNN, LSTM, BiLSTM, 트랜스포머를 벤치마크합니다. 단일 언어 마라티어 BERT가 다언어 BERT보다 우수하다는 점을 보이고, MahaBERT가 이진 정확도에서 최상, MahaRoBERTa가 4-클래스 정확도에서 최상을 달성합니다.

ABSTRACT

Social media platforms are used by a large number of people prominently to express their thoughts and opinions. However, these platforms have contributed to a substantial amount of hateful and abusive content as well. Therefore, it is important to curb the spread of hate speech on these platforms. In India, Marathi is one of the most popular languages used by a wide audience. In this work, we present L3Cube-MahaHate, the first major Hate Speech Dataset in Marathi. The dataset is curated from Twitter, annotated manually. Our dataset consists of over 25000 distinct tweets labeled into four major classes i.e hate, offensive, profane, and not. We present the approaches used for collecting and annotating the data and the challenges faced during the process. Finally, we present baseline classification results using deep learning models based on CNN, LSTM, and Transformers. We explore mono-lingual and multi-lingual variants of BERT like MahaBERT, IndicBERT, mBERT, and xlm-RoBERTa and show that mono-lingual models perform better than their multi-lingual counterparts. The MahaBERT model provides the best results on L3Cube-MahaHate Corpus. The data and models are available at https://github.com/l3cube-pune/MarathiNLP .

연구 동기 및 목표

• Create a large, manually annotated Marathi hate speech dataset from Twitter with fine-grained labels (Hate, Offensive, Profane, Not).
• Provide baseline benchmarks using CNN, LSTM, BiLSTM, and transformer models including monolingual and multilingual BERT variants.
• Compare mono-lingual Marathi BERT models with multilingual counterparts to assess performance advantages.
• Offer publicly available datasets and models for Marathi NLP and hate speech detection.

제안 방법

• Data collection from Twitter using Marathi hate-related keywords and over 150 bad words; manual annotation by native Marathi speakers into four classes.
• Dataset statistics with train/val/test splits and uniform class distribution to simplify modeling.
• Baseline experiments with CNN, LSTM, and BiLSTM using random, trainable, and non-trainable fastText embeddings.
• Transformer-based models including IndicBERT, mBERT, xlm-RoBERTa, MahaBERT, MahaALBERT, MahaRoBERTa; fine-tuning and evaluation on 2-class and 4-class tasks.
• Public release of MahaHate-BERT and MahaHate-MultiRoberta variants on model hubs; resources on GitHub.

실험 결과

연구 질문

• RQ1 How well do state-of-the-art deep learning architectures perform on the Marathi hate speech dataset for binary and four-class classification?
• RQ2 Do monolingual Marathi BERT models outperform multilingual BERT variants for Marathi hate speech detection?
• RQ3 Which transformer variants yield the best accuracy for 2-class and 4-class Marathi hate speech classification?
• RQ4 What is the impact of using trainable vs non-trainable embeddings on CNN/LSTM baselines?

주요 결과

• CNN/LSTM/BiLSTM baselines with non-trainable fastText embeddings yield competitive performance, often surpassing trainable variants.
• Monolingual Marathi BERT models outperform multilingual counterparts across tasks.
• MahaBERT achieves the best 2-class (binary) accuracy of 0.909, while MahaRoBERTa achieves the best 4-class accuracy of 0.803.
• Among transformer variants, MahaBERT and MahaRoBERTa deliver top results in binary and 4-class setups respectively.
• The best reported results are 2-class: 0.909 (MahaBERT); 4-class: 0.803 (MahaRoBERTa).