[Paper Review] Multilingual Alignment of Contextual Word Representations

Steven Cao, Nikita Kitaev+1 more|arXiv (Cornell University)|Feb 10, 2020

Topic ModelingComputer Science29 references152 citations

TL;DR

The paper introduces a contextual alignment procedure for multilingual BERT, showing improved zero-shot XNLI transfer and proposing a contextual word retrieval metric that correlates with downstream performance. It demonstrates significant gains with alignment, matching translate-train for Bulgarian and Greek, and analyzes strengths/weaknesses of alignment.

ABSTRACT

We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual BERT. In particular, after our proposed alignment procedure, BERT exhibits significantly improved zero-shot performance on XNLI compared to the base model, remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek. Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer. Using this word retrieval task, we also analyze BERT and find that it exhibits systematic deficiencies, e.g. worse alignment for open-class parts-of-speech and word pairs written in different scripts, that are corrected by the alignment procedure. These results support contextual alignment as a useful concept for understanding large multilingual pre-trained models.

Motivation & Objective

• Motivate understanding of multilingual BERT beyond raw zero-shot performance.
• Define and measure contextual alignment using a word retrieval task across parallel corpora.
• Propose a fine-tuning-based alignment method operating on multilingual BERT with parallel data.
• Evaluate alignment impact on zero-shot cross-lingual transfer (XNLI) and compare to baselines.
• Analyze linguistic factors affecting alignment, including POS classes and script differences.

Proposed method

• Define a contextual alignment objective with parallel corpora and word alignments.
• Use a CSLS-based similarity for retrieval but optimize a squared-distance loss with regularization.
• Regularize to keep embeddings close to initial pre-trained values while aligning non-English embeddings toward English.
• Train alignment jointly across multiple language pairs using parallel Europarl data.
• Compare against sentence-augmented fastText baselines and rotation-based contextual alignment methods.
• Evaluate zero-shot transfer on XNLI by fine-tuning on English MultiNLI and testing on Bulgarian, German, Greek, Spanish, and French.

Experimental results

Research questions

• RQ1 Can contextual alignment improve cross-lingual transfer for multilingual BERT beyond base performance?
• RQ2 How does the proposed alignment compare to rotation-based and non-contextual methods?
• RQ3 What data efficiency (parallel data amount) is required to achieve meaningful gains?
• RQ4 What are the linguistic factors (POS, lexical overlap, script differences) that influence alignment quality?

Key findings

• Contextual alignment substantially improves XNLI zero-shot accuracies across languages, with at least 1% gains and larger boosts for Bulgarian and Greek.
• Average zero-shot improvement over the base model is 2.78% when aligning with parallel data, closely approaching translate-train ceilings for Bulgarian and Greek.
• Fine-tuned contextual alignment outperforms fastText-based and rotation-based baselines, underscoring the value of context in alignment.
• The alignment correlates strongly with downstream transfer, validating contextual word retrieval as a predictive evaluation metric for cross-lingual performance.
• Alignment mitigates systematic deficiencies, notably improving open-class vs closed-class POS alignment and handling cross-script word pairs.
• With 50K parallel sentences per language pair, a large portion of gains are achieved; even 10K sentences yield improvements, highlighting data efficiency.