[Paper Review] Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages

Michael A. Hedderich, David Ifeoluwa Adelani+4 more|arXiv (Cornell University)|Oct 7, 2020

Natural Language Processing TechniquesComputer Science45 references49 citations

TL;DR

This paper studies transfer learning and distant supervision for multilingual transformer models (mBERT and XLM-RoBERTa) on three African languages (Hausa, isiXhosa, Yorùbá) for NER and news topic classification, showing that very small labeled datasets can achieve competitive performance in realistic low-resource settings, with caveats.

ABSTRACT

Multilingual transformer models like mBERT and XLM-RoBERTa have obtained great improvements for many NLP tasks on a variety of languages. However, recent works also showed that results from high-resource languages could not be easily transferred to realistic, low-resource scenarios. In this work, we study trends in performance for different amounts of available resources for the three African languages Hausa, isiXhosa and Yor\\`ub\\'a on both NER and topic classification. We show that in combination with transfer learning or distant supervision, these models can achieve with as little as 10 or 100 labeled sentences the same performance as baselines with much more supervised training data. However, we also find settings where this does not hold. Our discussions and additional experiments on assumptions such as time and hardware restrictions highlight challenges and opportunities in low-resource learning.

Motivation & Objective

• Motivate evaluation of multilingual transformers in realistic low-resource African languages.
• Assess transfer learning from high-resource languages to Hausa, isiXhosa, Yorùbá for NER and topic classification.
• Evaluate distant supervision as an alternative or complement to manual labeling in low-resource settings.
• Investigate practical considerations like time, hardware, and annotation effort in low-resource NLP.
• Provide publicly available datasets and insights to guide future low-resource multilingual NLP research.

Proposed method

• Evaluate mBERT and XLM-RoBERTa on NER and topic classification across Hausa, isiXhosa, and Yorùbá.
• Compare transformer models to traditional RNN-based baselines (GRU, LSTM-CNN-CRF, RCNN) using language-model pretraining.
• Study transfer learning by fine-tuning on high-resource English tasks (CoNLL03 NER, AG News) and evaluating zero-shot or few-shot transfer to target languages.
• Produce and assess distant supervision rules based on Wikidata entity lists and language-specific heuristics for NER and topic labels.
• Investigate label noise handling and the impact of combining distant supervision with noise-robust training.
• Discuss practical considerations like development sets, hardware constraints, and annotation time in low-resource scenarios.

Experimental results

Research questions

• RQ1 How do more complex transformer models compare to established RNNs in low-resource African languages?
• RQ2 How effective is transfer learning from high-resource languages to Hausa, isiXhosa, and Yorùbá for NER and topic classification?
• RQ3 Is distant supervision beneficial for these low-resource languages when used with transformer models?
• RQ4 What realistic assumptions (development sets, hardware, annotation time) affect low-resource multilingual NLP in practice?

Key findings

• Transformers often outperform RNN baselines in low-resource settings for NER on Hausa and Yorùbá and for isiXhosa in some settings, with CRFs still competitive in certain cases.
• Transfer learning from English can yield substantial gains; for NER, using as few as ten labeled target sentences can improve F1 by at least 10 points on a shared label set.
• Distant supervision provides useful gains when data is extremely scarce, e.g., Yorùbá NER with 100 labeled sentences matching 400 manually labeled sentences under certain noise-handling setups.
• Distant supervision alone yields modest F1 on test sets (e.g., 54% Hausa NER, 62% Yorùbá NER; 49% Hausa, 55% Yorùbá for topic classification), and its benefits depend on label noise handling and available manual data.
• Hardware and model size influence outcomes, with DistilBERT offering similar performance in some cases but reduced performance for NER with very few examples.
• Practical annotation time considerations show distant supervision and rule-based labeling can be feasible but require careful evaluation against manual labeling effort.