[Paper Review] An open access NLP dataset for Arabic dialects : Data collection, labeling, and model construction

ElMehdi Boujou, Hamza Chataoui+4 more|arXiv (Cornell University)|Feb 7, 2021

Natural Language Processing Techniques7 references17 citations

TL;DR

This paper introduces an open-access, multi-dialect, multi-topic NLP dataset of over 50,000 manually labeled Arabic social media tweets across five national dialects (Moroccan, Algerian, Tunisian, Egyptian, and Gulf). The dataset supports three key NLP tasks—dialect identification, topic classification, and sentiment analysis—using supervised learning models, with Naive Bayes achieving the highest F1-score (0.75) for dialect detection and Linear SVC performing best for topic detection (F1-micro: 0.84). The dataset is publicly available to accelerate research and innovation in low-resource Arabic dialect NLP.

ABSTRACT

Natural Language Processing (NLP) is today a very active field of research and innovation. Many applications need however big sets of data for supervised learning, suitably labelled for the training purpose. This includes applications for the Arabic language and its national dialects. However, such open access labeled data sets in Arabic and its dialects are lacking in the Data Science ecosystem and this lack can be a burden to innovation and research in this field. In this work, we present an open data set of social data content in several Arabic dialects. This data was collected from the Twitter social network and consists on +50K twits in five (5) national dialects. Furthermore, this data was labeled for several applications, namely dialect detection, topic detection and sentiment analysis. We publish this data as an open access data to encourage innovation and encourage other works in the field of NLP for Arabic dialects and social media. A selection of models were built using this data set and are presented in this paper along with their performances.

Motivation & Objective

• To address the scarcity of open, high-quality, labeled datasets for Arabic dialects (DA) in NLP, especially for low-resource variants beyond Egyptian and Gulf dialects.
• To collect and manually annotate a large-scale, multi-dialect, multi-topic social media dataset from Twitter to support diverse NLP applications.
• To provide a publicly available benchmark dataset that reduces the burden of data collection and labeling for researchers and developers in Arabic NLP.
• To evaluate baseline models across three core NLP tasks—dialect detection, topic classification, and sentiment analysis—on the proposed dataset.

Proposed method

• Data was collected from Twitter using multilingual hashtags and keywords to target five national Arabic dialects: Moroccan, Algerian, Tunisian, Egyptian, and Gulf.
• The dataset comprises 52,210 tweets after filtering, with manual labeling for three tasks: dialect detection, topic classification (Politics, Health, Social, Sport, Economics, and 'Other'), and sentiment analysis (Positive, Negative, Neutral).
• Labeling was performed by human annotators with inter-annotator agreement checks to ensure quality, and the dataset was published with full transparency and accessibility.
• Supervised machine learning models—Logistic Regression, SGD Classifier, Linear SVC, and Naive Bayes—were trained and evaluated using grid search and pipeline optimization.
• For imbalanced data, especially in topic detection (75% 'Other' category), under-sampling of the majority class was recommended to improve model generalization.
• Performance was evaluated using standard metrics: F1-score (micro), precision, recall, accuracy, and balanced accuracy across all tasks.

Experimental results

Research questions

• RQ1 Can a large-scale, multi-dialect, multi-topic Arabic social media dataset be effectively collected and manually annotated for NLP tasks?
• RQ2 How well do standard supervised machine learning models perform on dialect identification, topic classification, and sentiment analysis using this newly released dataset?
• RQ3 What are the performance trade-offs between different models (e.g., Logistic Regression vs. Naive Bayes) across the three NLP tasks on low-resource Arabic dialects?
• RQ4 To what extent does data imbalance—particularly in the 'Other' topic category—affect model performance, and how can it be mitigated?
• RQ5 Can this open-access dataset serve as a reliable baseline for future research and innovation in Arabic dialect NLP?

Key findings

• The Naive Bayes classifier achieved the highest F1-score (0.75) for dialect detection, outperforming Logistic Regression (0.72), SGD Classifier (0.73), and Linear SVC (0.75) in micro-F1.
• Linear Support Vector Classifier (Linear SVC) achieved the best F1-micro score of 0.84 for topic detection, followed by Logistic Regression (0.82), despite the high imbalance in the 'Other' class.
• For sentiment analysis, the SGD Classifier achieved the highest F1-micro score (0.74) and accuracy (0.77), slightly outperforming Logistic Regression and Linear SVC, while Naive Bayes showed the weakest performance (F1-micro: 0.67).
• The dataset contains 52,210 tweets labeled for sentiment, with 30,033 (57.5%) labeled as neutral, 15,385 (29.5%) as negative, and 6,792 (13%) as positive, reflecting a strong class imbalance.
• The 'Other' topic category accounted for 75% of topic-labeled tweets (37,313 out of 50,000), indicating a significant class imbalance that necessitates under-sampling for effective model training.
• The dataset is publicly available at [1] and includes all baseline models at [8], enabling reproducibility and benchmarking for future research in Arabic dialect NLP.