[Paper Review] Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT

Shijie Wu, Mark Dredze|arXiv (Cornell University)|Apr 19, 2019

Topic Modeling35 references59 citations

TL;DR

The paper investigates zero-shot cross-lingual transfer using multilingual BERT (mBERT) across five NLP tasks and 39 languages, showing competitive or state-of-the-art results without explicit cross-lingual signals and analyzing layer-wise behavior and subword sharing.

ABSTRACT

Pretrained contextual representation models (Peters et al., 2018; Devlin et al., 2018) have pushed forward the state-of-the-art on many NLP tasks. A new release of BERT (Devlin, 2018) includes a model simultaneously pretrained on 104 languages with impressive performance for zero-shot cross-lingual transfer on a natural language inference task. This paper explores the broader cross-lingual potential of mBERT (multilingual) as a zero shot language transfer model on 5 NLP tasks covering a total of 39 languages from various language families: NLI, document classification, NER, POS tagging, and dependency parsing. We compare mBERT with the best-published methods for zero-shot cross-lingual transfer and find mBERT competitive on each task. Additionally, we investigate the most effective strategy for utilizing mBERT in this manner, determine to what extent mBERT generalizes away from language specific features, and measure factors that influence cross-lingual transfer.

Motivation & Objective

• Assess whether mBERT learns a shared cross-lingual space enabling zero-shot transfer across diverse languages.
• Evaluate mBERT on five NLP tasks to compare with prior zero-shot methods.
• Identify effective fine-tuning and feature extraction strategies for cross-lingual transfer.
• Investigate how language-specific information is retained across layers and the impact of subword sharing on transfer.

Proposed method

• Use base cased mBERT (104 languages) and fine-tune on English data for 5 tasks (MLDoc, XNLI, NER, POS tagging, dependency parsing).
• Apply task-specific output layers and standard fine-tuning (3–4 epochs, learning rate ~3e-5) with Adam optimization.
• Experiment with freezing bottom layers (n in {0,3,6,9}) during fine-tuning and compare to a feature-based shallow combination of all layers (Feats).
• Measure cross-lingual transfer performance in zero-shot setting to 38 target languages per task; report UAS/LAS for parsing, F1 for NER, ACC for POS, accuracy for MLDoc and XNLI.
• Analyze language-identification accuracy per layer to assess language-specific information retention.
• Examine subword overlap across languages and its relation to transfer performance using WordPiece vocabulary.

Experimental results

Research questions

• RQ1 Does mBERT enable strong zero-shot cross-lingual transfer across a broad set of tasks without explicit cross-lingual supervision?
• RQ2 How does freezing lower layers vs. fine-tuning all parameters affect cross-lingual transfer performance across tasks?
• RQ3 Do different layers of mBERT retain language-specific information or abstract away language specifics while enabling cross-lingual transfer?
• RQ4 Does sharing subwords across languages correlate with improved transfer performance and can it indicate suitable source languages for transfer?

Key findings

• mBERT achieves competitive or state-of-the-art zero-shot cross-lingual performance across all five tasks, often outperforming prior cross-lingual embeddings.
• Fixing the bottom layers during fine-tuning generally improves performance, with the largest gains for document classification and NLI when using about six frozen layers and three for word-level tasks.
• Language identification tests show that language-specific information remains detectable in all layers, even as cross-lingual transfer improves.
• There is a strong positive correlation between subword overlap across languages and transfer performance, especially for type- and token-level overlap in four tasks (XNLI not included).
• In dependency parsing, mBERT surpasses a strong cross-lingual baseline by 7.3 UAS and 0.4 LAS on average without gold POS tags; with gold POS tags, gains further (1.6 UAS and 4.7 LAS on average).
• Substantial transfer differences are observed between closely related and distantly related languages, indicating room for improvement in distant language transfer.