[Paper Review] Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer

Iulia Turc, Kenton Lee+3 more|arXiv (Cornell University)|Jun 30, 2021

Topic ModelingComputer Science25 references37 citations

TL;DR

The paper shows that English is not always the best transfer language for zero-shot cross-lingual transfer; German and Russian often transfer more effectively across diverse target languages, and translating English training data into a better source language can improve zero-shot performance.

ABSTRACT

Despite their success, large pre-trained multilingual models have not completely alleviated the need for labeled data, which is cumbersome to collect for all target languages. Zero-shot cross-lingual transfer is emerging as a practical solution: pre-trained models later fine-tuned on one transfer language exhibit surprising performance when tested on many target languages. English is the dominant source language for transfer, as reinforced by popular zero-shot benchmarks. However, this default choice has not been systematically vetted. In our study, we compare English against other transfer languages for fine-tuning, on two pre-trained multilingual models (mBERT and mT5) and multiple classification and question answering tasks. We find that other high-resource languages such as German and Russian often transfer more effectively, especially when the set of target languages is diverse or unknown a priori. Unexpectedly, this can be true even when the training sets were automatically translated from English. This finding can have immediate impact on multilingual zero-shot systems, and should inform future benchmark designs.

Motivation & Objective

• Motivate the evaluation of transfer language effectiveness beyond English in zero-shot cross-lingual transfer.
• Quantify how non-English source languages transfer to diverse target languages using multilingual models.
• Assess whether machine-translated training data from English can outperform or match English-based transfer.
• Provide actionable guidance for data collection and benchmark design in multilingual NLP.

Proposed method

• Define zero-shot transferability metrics comparing source S to target T using relative abilities Z(S→T) = E(M^S, T) / E(M^T, T).
• Aggregate transferability over a target set L with Z(S→L) = (1/|L|) Σ_T∈L Z(S→T).
• Experiment with two pre-trained multilingual models (mBERT and mT5-base) on classification and QA tasks.
• Use monolingual fine-tuning in a single source language, then evaluate zero-shot performance on multiple targets.
• Control for data quality by comparing English-origin training data with machine-translated variants to assess translation effects.

Experimental results

Research questions

• RQ1 Is English the most effective source language for zero-shot cross-lingual transfer across diverse target languages?
• RQ2 Which non-English languages provide stronger cross-lingual transfer in standard benchmarks (XNLI, PAWS-X, XQuAD, TyDi QA)?
• RQ3 Does translating English training data into other languages improve zero-shot transfer compared to training directly in English?
• RQ4 Do results differ between encoder-only models (mBERT) and encoder-decoder models (mT5) across tasks?
• RQ5 How does transfer performance vary when target languages span different scripts and language families?

Key findings

• German and Russian often outperform English as transfer sources across XNLI and PAWS-X.
• Zero-shot advantage over English is observed for several non-English sources, e.g., de MT and ru MT show positive advantages on average.
• Translating English training data into stronger sources (e.g., German or Russian) can boost zero-shot transfer, even when MT data is used.
• For QA tasks with mT5, English can be the strongest source, but transfer from German/Russian still benefits certain targets (e.g., Thai).
• mBERT generally shows larger gaps between best and worst sources than mT5, and pre-training scale and strategy influence transfer dynamics.