[Paper Review] XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation

Yaobo Liang, Nan Duan+22 more|arXiv (Cornell University)|Apr 3, 2020

Natural Language Processing TechniquesComputer Science24 references67 citations

TL;DR

XGLUE introduces a cross-lingual benchmark with 11 tasks spanning understanding and generation, and evaluates strong baselines like Unicoder and XLM-R across 19 languages.

ABSTRACT

In this paper, we introduce XGLUE, a new benchmark dataset that can be used to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora and evaluate their performance across a diverse set of cross-lingual tasks. Comparing to GLUE(Wang et al., 2019), which is labeled in English for natural language understanding tasks only, XGLUE has two main advantages: (1) it provides 11 diversified tasks that cover both natural language understanding and generation scenarios; (2) for each task, it provides labeled data in multiple languages. We extend a recent cross-lingual pre-trained model Unicoder(Huang et al., 2019) to cover both understanding and generation tasks, which is evaluated on XGLUE as a strong baseline. We also evaluate the base versions (12-layer) of Multilingual BERT, XLM and XLM-R for comparison.

Motivation & Objective

• Provide a multilingual benchmark for training large-scale cross-lingual models using multilingual and bilingual corpora.
• Evaluate cross-lingual transfer capabilities across diverse tasks including understanding and generation.
• Extend a cross-lingual pre-trained model (Unicoder) to cover generation tasks and compare with baselines like M-BERT, XLM and XLM-R.
• Assess how task setup (pivot-language vs multi-language fine-tuning) and pre-training data affect cross-lingual performance.

Proposed method

• Pre-train Unicoder in a cross-lingual setting using MLM and TLM on multilingual and bilingual corpora.
• Extend Unicoder to generation by encoder-decoder architecture with shared BPE embeddings.
• Propose xDAE (multilingual denoising auto-encoding) and xFNP (multilingual future n-gram prediction) as pre-training objectives for generation.
• Use 11 tasks covering NER, POS, NC, MLQA, XNLI, PAWS-X, QADSM, WPR, QAM, QG, NTG to evaluate cross-lingual capabilities.
• Fine-tune pre-trained models on English labels only and evaluate on test sets in multiple languages.

Experimental results

Research questions

• RQ1 Can cross-lingual pre-trained models trained on multilingual/bilingual data generalize to a diverse set of tasks in many languages?
• RQ2 Does expanding pre-training to generation tasks improve cross-lingual generation and understanding performance?
• RQ3 What is the impact of pivot-language vs multi-language fine-tuning on cross-lingual transfer?
• RQ4 How do pre-training data size and language coverage influence cross-lingual transfer for both understanding and generation tasks?
• RQ5 Do generation-specific pre-training objectives (xDAE, xFNP) outperform pure understanding pre-training baselines on generation tasks?

Key findings

• Unicoder LC slightly outperforms M-BERT and XLM-R base on the 9 understanding tasks.
• Unicoder LC outperforms Unicoder SC due to larger pre-training corpus and use of TLM.
• Unicoder xDAE and Unicoder xFNP significantly outperform M-BERT and XLM-R base on the 2 generation tasks.
• Unicoder xFNP tends to perform slightly better than xDAE on generation tasks, though the comparison is not perfectly fair due to different strategies.
• Pre-training expansion to generation tasks is important for cross-lingual text generation capabilities.