[Paper Review] Cross-Lingual Ability of Multilingual BERT: An Empirical Study

K Karthikeyan, Zihan Wang+2 more|arXiv (Cornell University)|Dec 17, 2019

Topic ModelingComputer Science11 references172 citations

TL;DR

The paper investigates what drives cross-lingual transfer in Multilingual BERT (M-BERT) by probing bilingual BERT variants across linguistic properties, architecture, and learning objectives, finding that structural similarity and model depth are key while word-piece overlap and multi-head attention are less critical.

ABSTRACT

Recent work has exhibited the surprising cross-lingual abilities of multilingual BERT (M-BERT) -- surprising since it is trained without any cross-lingual objective and with no aligned data. In this work, we provide a comprehensive study of the contribution of different components in M-BERT to its cross-lingual ability. We study the impact of linguistic properties of the languages, the architecture of the model, and the learning objectives. The experimental study is done in the context of three typologically different languages -- Spanish, Hindi, and Russian -- and using two conceptually different NLP tasks, textual entailment and named entity recognition. Among our key conclusions is the fact that the lexical overlap between languages plays a negligible role in the cross-lingual success, while the depth of the network is an integral part of it. All our models and implementations can be found on our project page: http://cogcomp.org/page/publication_view/900 .

Motivation & Objective

• Assess what linguistic properties enable cross-lingual transfer in M-BERT across English and target languages (Spanish, Hindi, Russian).
• Evaluate the roles of model architecture (depth, attention heads, parameter count) on monolingual and cross-lingual performance.
• Examine input representations and learning objectives (NSP, language markers, tokenization) for cross-lingual learning.
• Differentiate effects of lexical overlap versus structural similarity on transfer.
• Develop methodologies to analyze language similarities by manipulating surface forms (e.g., Fake-English).

Proposed method

• Train bilingual BERT (B-BERT) variants on English and a target language using Wikipedia data.
• Evaluate cross-lingual transfer on XNLI (textual entailment) and NER using CO dataset configurations.
• Systematically vary word-piece overlap by introducing a Fake-English language with disjoint pieces.
• Control word-order similarity by permuting word order during pre-training.
• Analyze effects of depth, number of attention heads, and total parameter count on cross-lingual transfer.
• Test input representations (NSP, language identity markers, character/word-piece/word tokenization) and their impact on performance.

Experimental results

Research questions

• RQ1 How important is word-piece overlap for cross-lingual transfer in M-BERT compared to structural similarity between languages?
• RQ2 What is the relative contribution of model depth, attention heads, and total parameters to cross-lingual transfer?
• RQ3 Do learning objectives like NSP or language identity markers affect cross-lingual performance?
• RQ4 How do different input tokenizations (character, word-piece, word) influence cross-lingual abilities?

Key findings

• Cross-lingual transfer is possible even with no word-piece overlap; word-piece overlap contributes little to performance.
• Structural similarity, not surface overlap, drives cross-lingual transfer; word-order and higher-order language structure matter.
• Increasing model depth improves both monolingual and cross-lingual performance; total parameter count matters but depth is more influential.
• The number of attention heads is not crucial; a single head can yield satisfactory cross-lingual results.
• Next Sentence Prediction (NSP) hurts cross-lingual performance; language identity markers in input do not significantly affect cross-lingual transfer; tokenization with word-pieces or words outperforms character-level tokenization.