[Paper Review] Sexism detection: The first corpus in Algerian dialect with a code-switching in Arabic/ French and English

Imane Guellil, Ahsan Adeel+4 more|arXiv (Cornell University)|Apr 3, 2021

Hate Speech and Cyberbullying Detection55 references7 citations

TL;DR

This paper introduces the first publicly available hate speech corpus in Algerian Darija with code-switching between Arabic, French, and English, specifically targeting sexism against women on social media. Annotators labeled 5,000 YouTube comments, resulting in a balanced corpus of 1,798 comments and an unbalanced one of 3,798; deep learning models, especially CNN, achieved an F1-score of 86% on the unbalanced corpus, outperforming LSTM and Bi-LSTM.

ABSTRACT

In this paper, an approach for hate speech detection against women in Arabic community on social media (e.g. Youtube) is proposed. In the literature, similar works have been presented for other languages such as English. However, to the best of our knowledge, not much work has been conducted in the Arabic language. A new hate speech corpus (Arabic\_fr\_en) is developed using three different annotators. For corpus validation, three different machine learning algorithms are used, including deep Convolutional Neural Network (CNN), long short-term memory (LSTM) network and Bi-directional LSTM (Bi-LSTM) network. Simulation results demonstrate the best performance of the CNN model, which achieved F1-score up to 86\% for the unbalanced corpus as compared to LSTM and Bi-LSTM.

Motivation & Objective

• To address the lack of annotated datasets for hate speech detection in Arabic, particularly in Algerian Darija with code-switching.
• To develop a manually annotated corpus of online comments targeting women on social media platforms like YouTube.
• To evaluate multiple machine learning and deep learning models for sexism detection in low-resource, code-switched Arabic dialect contexts.
• To establish a benchmark for future research in Arabic hate speech detection, especially for gendered hate speech.
• To improve detection performance through future integration of transliteration and multilingual language identification.

Proposed method

• Data was automatically collected from YouTube comments related to women, resulting in 5,000 initial comments.
• Three native Algerian Darija speakers annotated the comments as hateful or non-hateful using a standardized guideline.
• A subset of 3,798 comments with full annotator agreement was used to construct an unbalanced corpus (3,006 non-hateful, 792 hateful).
• A balanced subset of 1,798 comments was randomly selected from the agreement set for comparative evaluation.
• Multiple machine learning models (LSVC, GNB, LR, SGD, RF) and deep learning models (CNN, LSTM, Bi-LSTM) were trained and evaluated on the corpus.
• Word2Vec and FastText embeddings were used to represent text sequences in the deep learning models.

Experimental results

Research questions

• RQ1 What is the performance of different machine learning and deep learning models in detecting sexism in Algerian Darija with code-switching in Arabic, French, and English?
• RQ2 How does the choice of corpus (balanced vs. unbalanced) affect model performance in sexism detection?
• RQ3 Can deep learning models like CNN, LSTM, and Bi-LSTM outperform traditional machine learning models on this low-resource, code-switched Arabic dialect corpus?
• RQ4 What improvements can be achieved through preprocessing techniques such as transliteration and language identification?
• RQ5 How can a manually annotated, publicly available corpus support future research in Arabic hate speech detection?

Key findings

• The CNN model achieved the highest F1-score of 86% on the unbalanced corpus, outperforming both LSTM and Bi-LSTM models.
• The Bi-LSTM model achieved an F1-score of 85% on the unbalanced corpus, demonstrating strong performance for sequence modeling in code-switched text.
• The CNN model achieved an F1-score of 82% on the balanced corpus, indicating robustness despite class imbalance.
• The LSVC model achieved an F1-score of 80% on the unbalanced corpus, showing competitive performance among traditional classifiers.
• The GNB and LR models showed lower performance, with F1-scores of 61% and 72% respectively, indicating limited effectiveness for this task.
• The study identifies transliteration and multilingual language identification as key future improvements to enhance model accuracy.