[Paper Review] How multilingual is Multilingual BERT?

Telmo Pires, Eva Schlinger+1 more|arXiv (Cornell University)|Jun 4, 2019

Natural Language Processing Techniques16 references139 citations

TL;DR

The paper demonstrates that Multilingual BERT (M-BERT) trained on 104 languages shows strong zero-shot cross-lingual transfer, revealing a multilingual representation that extends beyond simple vocabulary overlap, with varying transfer success across language similarity and scripts.

ABSTRACT

In this paper, we show that Multilingual BERT (M-BERT), released by Devlin et al. (2018) as a single language model pre-trained from monolingual corpora in 104 languages, is surprisingly good at zero-shot cross-lingual model transfer, in which task-specific annotations in one language are used to fine-tune the model for evaluation in another language. To understand why, we present a large number of probing experiments, showing that transfer is possible even to languages in different scripts, that transfer works best between typologically similar languages, that monolingual corpora can train models for code-switching, and that the model can find translation pairs. From these results, we can conclude that M-BERT does create multilingual representations, but that these representations exhibit systematic deficiencies affecting certain language pairs.

Motivation & Objective

• Investigate how well M-BERT generalizes across languages without explicit cross-lingual training.
• Examine whether cross-language transfer relies on lexical (vocabulary) overlap or deeper multilingual representations.
• Analyze transfer across different scripts and code-switching scenarios.
• Probe the influence of typological similarity on cross-lingual generalization.
• Characterize the structure of M-BERT's multilingual feature space.

Proposed method

• Use M-BERT pre-trained on 104 languages with a shared word piece vocabulary.
• Fine-tune on task data in one language and evaluate zero-shot in other languages for NER and POS tagging.
• Compare zero-shot performance to English-centered models to assess reliance on vocabulary overlap.
• Vary language scripts and typological features to study cross-script and cross-language transfer.
• Conduct probing analyses to map the geometry of M-BERT's multilingual feature space across languages.

Experimental results

Research questions

• RQ1 Does M-BERT enable zero-shot cross-lingual transfer across languages with different scripts?
• RQ2 To what extent does lexical overlap drive cross-lingual transfer, and can M-BERT transfer beyond vocabulary memorization?
• RQ3 How does typological similarity affect zero-shot transfer performance?
• RQ4 Can M-BERT generalize to code-switched and transliterated text without multilingual pretraining?
• RQ5 What is the structure of M-BERT's multilingual representation space across languages?

Key findings

• M-BERT achieves strong zero-shot NER and POS performance across language pairs, including languages with different scripts.
• Zero-shot NER transfer is largely independent of word-piece overlap, indicating a multilingual representation beyond lexical memorization.
• POS transfer shows high accuracy across European language pairs (>80% in many cases) and varies with script and typology.
• Transfer across scripts is possible (e.g., Urdu to Hindi) but not uniformly; typological similarity improves transfer performance.
• Code-switching and transliteration pose challenges; transliteration transfer is weaker without transliteration-aware training.
• Probing suggests a language-agnostic subspace in lower layers of M-BERT, with language-specific signals emerging in higher layers.
• The authors conclude that M-BERT learns multilingual representations but with systematic deficiencies for certain language pairs, indicating a potential need for explicit multilingual training objectives to improve cross-language generalization.