[Paper Review] DiaNet: BERT and Hierarchical Attention Multi-Task Learning of Fine-Grained Dialect

Muhammad Abdul-Mageed, Chiyu Zhang+3 more|arXiv (Cornell University)|Oct 31, 2019

Natural Language Processing Techniques37 references4 citations

TL;DR

This paper introduces DiaNet, a hierarchical attention multi-task learning model combined with BERT for fine-grained Arabic dialect identification across 21 Arab countries, including city- and state-level micro-dialects. Using a large-scale Twitter dataset of ~6 billion tweets, the approach achieves state-of-the-art performance, with 85.46% F1 on external data and strong user-level dialect detection using only 100 tweets per user.

ABSTRACT

Prediction of language varieties and dialects is an important language processing task, with a wide range of applications. For Arabic, the native tongue of ~ 300 million people, most varieties remain unsupported. To ease this bottleneck, we present a very large scale dataset covering 319 cities from all 21 Arab countries. We introduce a hierarchical attention multi-task learning (HA-MTL) approach for dialect identification exploiting our data at the city, state, and country levels. We also evaluate use of BERT on the three tasks, comparing it to the MTL approach. We benchmark and release our data and models.

Motivation & Objective

• To address the lack of large-scale, fine-grained Arabic dialect datasets for NLP, especially at the city and state levels.
• To develop a multi-task learning framework that jointly predicts dialect at hierarchical levels: city, state, and country.
• To evaluate the effectiveness of BERT in low-resource, fine-grained dialect classification tasks.
• To benchmark and release a large-scale, automatically and manually annotated Arabic dialect dataset covering 319 cities and all 21 Arab countries.

Proposed method

• The authors collect ~6 billion tweets from 2.7 million Twitter users across the Arab world using geolocation metadata from user profiles.
• They use the Nominatim geocoder to automatically assign city and country labels to users, creating a large-scale weakly-supervised dataset.
• A novel hierarchical attention multi-task learning (HA-MTL) architecture shares a BiGRU encoder across three tasks: city, state, and country classification, with multi-head attention at each level.
• The model is trained with joint supervision at all three levels, allowing knowledge transfer between coarse and fine-grained dialect prediction.
• BERT is fine-tuned on the same tasks to compare its performance against the HA-MTL model, with both models evaluated on zero-shot and few-shot settings.
• Manual validation is performed on ~2 million tweets from ~5,000 users to ensure data quality and dialect accuracy.

Experimental results

Research questions

• RQ1 Can location-based geocoding serve as an effective surrogate for dialect labeling in low-resource, fine-grained dialect classification?
• RQ2 How effective is hierarchical attention multi-task learning in capturing hierarchical linguistic variation across city, state, and country levels in Arabic dialects?
• RQ3 Does BERT outperform traditional sequence models in fine-grained dialect identification tasks with limited labeled data?
• RQ4 To what extent can a model trained on tweet-level labels generalize to user-level dialect detection using only 100–500 tweets per user?

Key findings

• The HA-MTL model achieves an F1 score of 85.464% on external test data, significantly outperforming prior work, including the SHAMI system which achieved 71.000% F1.
• With 100 tweets per user, the best BERT model reaches 65.171% accuracy in user-level dialect detection, improving to 66.787% with 500 tweets.
• The model maintains high coverage, as 94.85% of the 233,105 automatically tagged users have at least 100 tweets, enabling practical deployment.
• The proposed dataset of ~6 billion tweets and ~2 million manually labeled tweets is the largest and most fine-grained Arabic dialect dataset to date, covering 319 cities across 21 countries.
• BERT fine-tuned on the HA-MTL framework achieves state-of-the-art results on multiple external benchmarks, demonstrating its effectiveness in low-resource dialect NLP.
• The hierarchical multi-task structure enables knowledge transfer, improving performance on low-resource city-level classification by leveraging supervision from higher-level state and country tasks.