[Paper Review] Arabic Dialect Identification Using BERT-Based Domain Adaptation

Ahmad Beltagy, Abdelrahman Wael+1 more|arXiv (Cornell University)|Nov 13, 2020

Natural Language Processing Techniques12 references4 citations

TL;DR

This paper proposes a BERT-based domain adaptation approach for Arabic dialect identification using AraBERT fine-tuned on unlabeled Twitter data. By applying extensive text cleaning, data augmentation, and masked language modeling fine-tuning, the method achieves a 23.09% F1 macro score on the NADI shared task, ranking 4th among 21 dialects with a robust, semi-supervised deep learning framework.

ABSTRACT

Arabic is one of the most important and growing languages in the world. With the rise of social media platforms such as Twitter, Arabic spoken dialects have become more in use. In this paper, we describe our approach on the NADI Shared Task 1 that requires us to build a system to differentiate between different 21 Arabic dialects, we introduce a deep learning semi-supervised fashion approach along with pre-processing that was reported on NADI shared Task 1 Corpus. Our system ranks 4th in NADI's shared task competition achieving a 23.09% F1 macro average score with a simple yet efficient approach to differentiating between 21 Arabic Dialects given tweets.

Motivation & Objective

• To address the challenge of identifying 21 Arabic dialects in social media text, particularly given data imbalance and noisy samples.
• To improve classification performance using a semi-supervised deep learning approach with pre-trained BERT models.
• To mitigate the impact of noisy data, such as Quranic verses and English words, through targeted text preprocessing.
• To enhance model generalization by fine-tuning AraBERT on unlabeled Twitter data via masked language modeling for domain adaptation.
• To explore hybrid architectures combining BERT with naive Bayes for minority dialects, aiming to improve performance on low-resource classes.

Proposed method

• Preprocessing involved removing URLs, mentions, punctuation, English characters, emojis, and tashkil using regex and PyArabic, followed by normalization of elongated characters.
• Word segmentation was applied using Farasa Segmenter to separate prefixes and suffixes from root words, enhancing morphological representation.
• Data augmentation was performed via upsampling minority classes to address class imbalance in the training set.
• AraBERT was fine-tuned on the unlabeled NADI Twitter corpus using masked language modeling (MLM) to adapt the pre-trained model to the target domain.
• A hybrid model combined AraBERT with naive Bayes, where AraBERT first classified tweets into majority or minority dialect groups, and naive Bayes further classified minority-class predictions.
• The final model used a single transformer head with domain-adapted AraBERT, trained end-to-end on cleaned and augmented data, achieving state-of-the-art performance on the NADI benchmark.

Experimental results

Research questions

• RQ1 How effective is domain adaptation via masked language modeling fine-tuning on unlabeled Twitter data for Arabic dialect identification?
• RQ2 To what extent does text preprocessing and data augmentation improve performance on low-resource Arabic dialects?
• RQ3 Can a hybrid model combining BERT and naive Bayes outperform a pure BERT-based classifier for minority dialect classes?
• RQ4 How do noise factors like Quranic verses and English words affect model performance, and can preprocessing mitigate their impact?
• RQ5 What is the contribution of each component (cleaning, upsampling, MLM fine-tuning) to the overall F1 score improvement?

Key findings

• The baseline AraBERT model without preprocessing or data augmentation achieved a dev-set F1 score of 18.6%, indicating significant room for improvement.
• Text preprocessing alone improved the F1 score to 20.54%, demonstrating the effectiveness of cleaning noisy and inconsistent linguistic features.
• Combining preprocessing with upsampling increased the F1 score to 21.33%, showing that addressing class imbalance helps minority dialect recognition.
• Fine-tuning AraBERT on unlabeled Twitter data via masked language modeling yielded the largest improvement, raising the F1 score to 24.43% on the dev set.
• The final model achieved a test-set F1 macro score of 23.09%, ranking 4th in the NADI shared task, validating the effectiveness of domain adaptation.
• The hybrid AraBERT + naive Bayes model achieved 22.3% F1, which was less effective than the pure AraBERT approach, suggesting that simpler models outperform complex ensembles in this setting.