[Paper Review] XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization

Junjie Hu, Sebastian Ruder+4 more|arXiv (Cornell University)|Mar 24, 2020

Topic Modeling59 references299 citations

TL;DR

XTREME introduces a broad, zero-shot cross-lingual benchmark spanning 40 languages and 9 tasks to evaluate multilingual representations and transfer learning, revealing sizable cross-lingual gaps especially in syntactic and sentence retrieval tasks.

ABSTRACT

Much recent progress in applications of machine learning models to NLP has been driven by benchmarks that evaluate models across a wide variety of tasks. However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing. To this end, we introduce the Cross-lingual TRansfer Evaluation of Multilingual Encoders XTREME benchmark, a multi-task benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models, particularly on syntactic and sentence retrieval tasks. There is also a wide spread of results across languages. We release the benchmark to encourage research on cross-lingual learning methods that transfer linguistic knowledge across a diverse and representative set of languages and tasks.

Motivation & Objective

• Motivate the need for a comprehensive cross-lingual evaluation benchmark beyond English-centered tasks.
• Provide a diverse, typologically broad set of languages and tasks to assess cross-lingual transfer capabilities.
• Promote standardized evaluation and baselines to advance multilingual representation learning.
• Analyze limitations of current state-of-the-art cross-lingual models across languages and tasks.

Proposed method

• Define the Cross-lingual Transfer Evaluation of Multilingual Encoders (xtreme) benchmark with 40 languages and 9 tasks.
• Adopt zero-shot cross-lingual transfer where training data is English-only while testing on target languages.
• Assemble tasks spanning classification, structured prediction, and QA to test meaning transfer at multiple linguistic levels.
• Provide pseudo (translated) test sets for diagnostics to cover all languages and enable broader analyses.
• Evaluate strong baselines (mBERT, XLM, XLM-R, MMTE) and translation-based approaches, releasing code and a leaderboard.
• Analyze correlations between performance and pretraining data size, language family, and script to understand transfer dynamics.

Experimental results

Research questions

• RQ1 How well do current multilingual representations transfer to 40 typologically diverse languages across 9 tasks in a zero-shot setting?
• RQ2 What are the main cross-lingual transfer gaps and how do they vary by task and language family or script?
• RQ3 Do translation-based augmentation or in-language training data improve cross-lingual transfer relative to zero-shot transfer?
• RQ4 How does model performance correlate with pretraining data size and language characteristics (family, script)?
• RQ5 What diagnostics can reveal limitations of state-of-the-art cross-lingual models across diverse languages?

Key findings

• Zero-shot transfer models approach human performance in English but show sizable drops in other languages, especially on syntactic and sentence retrieval tasks.
• XLM-R Large generally outperforms mBERT and other baselines in zero-shot transfer, with notable gains on XQuAD and MLQA but limited gains on structured prediction tasks.
• Translation-based baselines (translate-train, translate-test) provide substantial gains, often reducing the cross-lingual transfer gap across tasks.
• In-language training data can outperform zero-shot transfer on several tasks, but zero-shot methods still compete well on complex QA tasks when English data are plentiful.
• Cross-lingual transfer correlates with pretraining data size in many languages, with stronger effects in Indo-European languages and weaker effects in Sino-Tibetan, Japonic, Koreanic, and Niger-Congo families.
• There remains a substantial transfer gap across languages and tasks, highlighting room for improvement in cross-lingual transfer methods.