[论文解读] Pre-Training BERT on Arabic Tweets: Practical Considerations

Ahmed Abdelalí, Sabit Hassan+3 more|arXiv (Cornell University)|Feb 21, 2021

Natural Language Processing Techniques参考文献 24被引用 83

一句话总结

The paper analyzes training BERT models from scratch on Arabic tweets, comparing data sources, segmentation strategies, and training regimes, and releases QARiB checkpoints with in-depth evaluation across multiple tasks.

ABSTRACT

Pretraining Bidirectional Encoder Representations from Transformers (BERT) for downstream NLP tasks is a non-trival task. We pretrained 5 BERT models that differ in the size of their training sets, mixture of formal and informal Arabic, and linguistic preprocessing. All are intended to support Arabic dialects and social media. The experiments highlight the centrality of data diversity and the efficacy of linguistically aware segmentation. They also highlight that more data or more training step do not necessitate better models. Our new models achieve new state-of-the-art results on several downstream tasks. The resulting models are released to the community under the name QARiB.

研究动机与目标

• Assess when from-scratch pre-training is warranted for Arabic BERT in tweet domains.
• Determine how data size, data mix (formal + informal), and segmentation affect performance.
• Evaluate language-specific vs. language-agnostic tokenization approaches on Arabic data.
• Compare QARiB models to existing Arabic BERT variants across multiple NLP tasks.
• Provide pretrained checkpoints to accelerate research and downstream fine-tuning.

提出的方法

• Train five QARiB models from scratch using varying data sources and segmentation (Farasa vs. no Farasa) and data sizes (10M–330M tweets).
• Use BPE-based tokenization with language-agnostic segmentation and compare to Farasa-segmented variants.
• Pretrain models with 15% masking on a single-task objective (masked language modeling) on TPU hardware to reduce training time.
• Evaluate models on a battery of tasks (NER, emotion, offensive language detection, dialect identification, sentiment) using task-specific datasets.
• Compare with three Arabic and multilingual baselines (AraBERTv0.1/v1, ArabicBERT, mBERT) at corresponding checkpoints.

实验结果

研究问题

• RQ1 What data scale is required to build effective Arabic BERT models for Twitter data?
• RQ2 Does mixing formal Arabic with informal tweets improve downstream performance compared to tweets alone?
• RQ3 Does language-specific segmentation (e.g., Farasa) meaningfully improve results over BPE-only tokenization for Arabic tweets?
• RQ4 Is pre-training from scratch more effective than fine-tuning existing models when data is in-domain (tweets), considering vocabulary coverage?
• RQ5 How should one determine optimal training checkpoints for various tasks rather than relying on loss alone?

主要发现

模型	AJGT	情感	命名实体识别	冒犯语言	QADI
QARiB10	92.2	43.6	61.3	88.5	60.1
QARiB25	93.3	44.7	63.8	90.0	60.7
QARiB25_mix	93.3	46.8	64.4	89.5	60.9
QARiB25_mix_far	93.3	45.2	69.1	89.0	61.3
QARiB60_mix	93.3	46.1	63.0	90.0	61.4
AraBERTv0.1	90.8	43.9	65.0	88.1	59.9
AraBERTv1	93.6	42.4	66.6	89.0	59.9
ArabicBERT	83.3	41.7	64.0	88.2	61.7
mBERT	86.6	27.9	49.4	83.1	57.8

• Increasing data from 10M to 25M tweets improves performance, but more data beyond 25M shows diminishing returns in some cases.
• Mixing tweets with formal Arabic data outperforms tweets-only training on downstream tweet tasks.
• Linguistically motivated segmentation (Farasa) yields substantial gains on some tasks, aligning with results for AraBERT variants.
• Checkpoints matter; more training steps do not guarantee better results, and multi-task evaluation across tasks helps identify the best checkpoint.
• QARiB models with mixed data and Farasa segmentation (e.g., QARiB25_mix_far) achieve strong results, often outperforming mBERT and comparable/stronger than AraBERT and ArabicBERT on several tasks.