[Paper Review] InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

Zewen Chi, Li Dong+8 more|arXiv (Cornell University)|Jul 15, 2020

Topic Modeling43 references77 citations

TL;DR

InfoXLM formulates cross-lingual pre-training as mutual information maximization across multilingual views and introduces XlCo, a cross-lingual contrast task, improving transferability by leveraging monolingual and parallel data.

ABSTRACT

In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts. The unified view helps us to better understand the existing methods for learning cross-lingual representations. More importantly, inspired by the framework, we propose a new pre-training task based on contrastive learning. Specifically, we regard a bilingual sentence pair as two views of the same meaning and encourage their encoded representations to be more similar than the negative examples. By leveraging both monolingual and parallel corpora, we jointly train the pretext tasks to improve the cross-lingual transferability of pre-trained models. Experimental results on several benchmarks show that our approach achieves considerably better performance. The code and pre-trained models are available at https://aka.ms/infoxlm.

Motivation & Objective

• Provide a unified information-theoretic view of cross-lingual pre-training tasks (MMLM, TLM) and their cross-lingual effects.
• Introduce a new pre-training task XlCo that maximizes sequence-level mutual information between translation pairs.
• Develop InfoXLM to jointly leverage monolingual and parallel corpora for better cross-lingual transfer.
• Evaluate InfoXLM on cross-lingual natural language inference, question answering, and sentence retrieval.

Proposed method

• Formulate cross-lingual pre-training as maximizing mutual information between multilingual-multi-granularity texts.
• Reinterpret MMLM as maximizing I(c1; x1) and cross-language anchors I(x1; x2) to encourage cross-lingual correlations.
• Explain TLM as maximizing I(c1, c2; x1) and thus leveraging cross-lingual context.
• Propose XlCo to maximize I(c1; c2) using InfoNCE with a momentum-contrast setup and a large negative set.
• Apply mixup contrast and contrast on a universal layer to improve cross-lingual transferability.
• Pre-train InfoXLM by jointly optimizing MMLM, TLM, and XlCo with English pivot alignment and multilingual sampling.

Experimental results

Research questions

• RQ1 Can an information-theoretic framework unify existing cross-lingual pre-training tasks and explain their cross-lingual transfer effects?
• RQ2 Does introducing a cross-lingual contrast task (XlCo) improve cross-lingual alignment and downstream transfer compared to MMLM/TLM alone?
• RQ3 What is the impact of performing XlCo on a universal layer versus the last layer?
• RQ4 How does mixup contrast and momentum contrast contribute to cross-lingual transfer and representation quality?
• RQ5 Does InfoXLM achieve better performance on XNLI, MLQA, and cross-lingual sentence retrieval across multiple languages?

Key findings

• InfoXLM outperforms strong baselines on XNLI, MLQA, and cross-lingual retrieval tasks.
• Cross-lingual contrast (XlCo) improves zero-shot transfer and cross-lingual alignment, especially when used on the universal layer.
• XlCo combined with TLM yields stronger cross-lingual transfer than either task alone, and momentum contrast enhances representations.
• InfoXLM reduces cross-lingual transfer gaps between English and other languages on MLQA and XNLI.
• InfoXLM base and large models achieve competitive or superior results to XLM-R and related baselines across multiple benchmarks.