[論文レビュー] Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer

Iulia Turc, Kenton Lee+3 more|arXiv (Cornell University)|Jun 30, 2021

Topic Modeling参考文献 25被引用数 37

ひとこと要約

この論文は、ゼロショットのクロス言 lingual転送において英語が必ずしも最良の転送言語ではないことを示している。ドイツ語とロシア語は多様なターゲット言語に対してしばしばより効果的に転送され、英語の訓練データをより良いソース言語へ翻訳することはゼロショットの性能を改善できる。

ABSTRACT

Despite their success, large pre-trained multilingual models have not completely alleviated the need for labeled data, which is cumbersome to collect for all target languages. Zero-shot cross-lingual transfer is emerging as a practical solution: pre-trained models later fine-tuned on one transfer language exhibit surprising performance when tested on many target languages. English is the dominant source language for transfer, as reinforced by popular zero-shot benchmarks. However, this default choice has not been systematically vetted. In our study, we compare English against other transfer languages for fine-tuning, on two pre-trained multilingual models (mBERT and mT5) and multiple classification and question answering tasks. We find that other high-resource languages such as German and Russian often transfer more effectively, especially when the set of target languages is diverse or unknown a priori. Unexpectedly, this can be true even when the training sets were automatically translated from English. This finding can have immediate impact on multilingual zero-shot systems, and should inform future benchmark designs.

研究の動機と目的

• Motivate the evaluation of transfer language effectiveness beyond English in zero-shot cross-lingual transfer.
• Quantify how non-English source languages transfer to diverse target languages using multilingual models.
• Assess whether machine-translated training data from English can outperform or match English-based transfer.
• Provide actionable guidance for data collection and benchmark design in multilingual NLP.

提案手法

• Define zero-shot transferability metrics comparing source S to target T using relative abilities Z(S→T) = E(M^S, T) / E(M^T, T).
• Aggregate transferability over a target set L with Z(S→L) = (1/|L|) Σ_T∈L Z(S→T).
• Experiment with two pre-trained multilingual models (mBERT and mT5-base) on classification and QA tasks.
• Use monolingual fine-tuning in a single source language, then evaluate zero-shot performance on multiple targets.
• Control for data quality by comparing English-origin training data with machine-translated variants to assess translation effects.]
• research_questions:[

実験結果

リサーチクエスチョン

• RQ1 Is English the most effective source language for zero-shot cross-lingual transfer across diverse target languages?
• RQ2 Which non-English languages provide stronger cross-lingual transfer in standard benchmarks (XNLI, PAWS-X, XQuAD, TyDi QA)?
• RQ3 Does translating English training data into other languages improve zero-shot transfer compared to training directly in English?
• RQ4 Do results differ between encoder-only models (mBERT) and encoder-decoder models (mT5) across tasks?
• RQ5 How does transfer performance vary when target languages span different scripts and language families?

主な発見

• German and Russian often outperform English as transfer sources across XNLI and PAWS-X.
• Zero-shot advantage over English is observed for several non-English sources, e.g., de MT and ru MT show positive advantages on average.
• Translating English training data into stronger sources (e.g., German or Russian) can boost zero-shot transfer, even when MT data is used.
• For QA tasks with mT5, English can be the strongest source, but transfer from German/Russian still benefits certain targets (e.g., Thai).
• mBERT generally shows larger gaps between best and worst sources than mT5, and pre-training scale and strategy influence transfer dynamics.