[Paper Review] BERT-Based Arabic Social Media Author Profiling

Chiyu Zhang, Muhammad Abdul-Mageed|arXiv (Cornell University)|Sep 9, 2019

Authorship Attribution and Profiling40 references9 citations

TL;DR

This paper proposes BERT-based models for Arabic social media author profiling, focusing on age, dialect, and gender prediction using fine-tuned multilingual BERT and in-house data augmentation. The best performance—54.72% accuracy for age, 93.75% for dialect, 81.67% for gender, and 40.97% joint accuracy—was achieved through majority voting of multiple BERT models trained under different data conditions, demonstrating the effectiveness of data augmentation and ensemble learning in low-resource Arabic NLP tasks.

ABSTRACT

We report our models for detecting age, language variety, and gender from social media data in the context of the Arabic author profiling and deception detection shared task (APDA). We build simple models based on pre-trained bidirectional encoders from transformers (BERT). We first fine-tune the pre-trained BERT model on each of the three datasets with shared task released data. Then we augment shared task data with in-house data for gender and dialect, showing the utility of augmenting training data. Our best models on the shared task test data are acquired with a majority voting of various BERT models trained under different data conditions. We acquire 54.72% accuracy for age, 93.75% for dialect, 81.67% for gender, and 40.97% joint accuracy across the three tasks.

Motivation & Objective

• To develop robust BERT-based models for Arabic author profiling on social media data, focusing on age, dialect, and gender prediction.
• To evaluate the impact of in-house labeled data augmentation on model performance for dialect and gender classification tasks.
• To investigate the effectiveness of ensemble learning via majority voting across multiple BERT models trained under varying data conditions.
• To address the challenge of low-resource author profiling in Arabic, particularly for underrepresented dialects and gender categories.
• To contribute to the APDA shared task by submitting competitive models with improved generalization through data and model ensembling.

Proposed method

• Fine-tuned multilingual BERT-Base cased model on shared task data for each of the three classification tasks: age, dialect, and gender.
• Introduced in-house labeled datasets—1,100 users for gender and 20,000 tweets per dialect class for data augmentation, combined with official training data to form extended datasets.
• Trained BERT models on both original shared task data and extended data (BERT_EXT) to improve generalization and reduce overfitting.
• Employed a two-stage pipeline: first predicting labels at the tweet level, then aggregating predictions at the user level by majority vote per user.
• Used majority voting across three distinct model configurations (first submission, second submission with DEV data, and user-level BERT) for final test predictions.
• Applied standard training protocols: sequence length of 50 tokens, batch size of 32, initial learning rate of 2e-5, and 15 training epochs with early stopping on validation set.

Experimental results

Research questions

• RQ1 How effective is BERT fine-tuning for Arabic author profiling tasks, particularly for age, dialect, and gender prediction?
• RQ2 To what extent does in-house data augmentation improve model performance on low-resource Arabic dialect and gender classification tasks?
• RQ3 Can ensemble learning via majority voting across multiple BERT models trained under different data conditions outperform individual models?
• RQ4 How does the integration of validation data (DEV) into training affect model generalization on the official test set?
• RQ5 What is the joint performance of a multi-task author profiling system on age, dialect, and gender prediction in the APDA shared task?

Key findings

• The best-performing model, achieved through majority voting of three distinct BERT configurations, obtained 54.72% accuracy on the age classification task.
• For dialect identification, the final submission achieved 93.75% accuracy, significantly outperforming the initial BERT model (93.33%) and the BERT_EXT model (95.56%) on the test set.
• Gender classification reached 81.67% accuracy in the final submission, surpassing the initial BERT model (77.08%) and the BERT_EXT model (84.00%) on the test set.
• The joint prediction accuracy across all three tasks reached 40.97%, indicating strong performance on the multi-label evaluation metric used in the shared task.
• Data augmentation with in-house datasets improved performance for both dialect and gender tasks, with BERT_EXT models showing consistent gains over baseline BERT models.
• The majority voting strategy proved highly effective, as it maintained or improved performance across all three tasks compared to individual model submissions.