[Paper Review] XNLI: Evaluating Cross-lingual Sentence Representations

Alexis Conneau, Guillaume Lample+5 more|arXiv (Cornell University)|Sep 13, 2018

Topic Modeling49 references152 citations

TL;DR

XNLI extends MultiNLI to 15 languages to evaluate cross-lingual sentence representations and multilingual transfer, comparing translation baselines with multilingual encoders and alignment-based methods.

ABSTRACT

State-of-the-art natural language processing systems rely on supervision in the form of annotated data to learn competent models. These models are generally trained on data in a single language (usually English), and cannot be directly used beyond that language. Since collecting data in every language is not realistic, there has been a growing interest in cross-lingual language understanding (XLU) and low-resource cross-language transfer. In this work, we construct an evaluation set for XLU by extending the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 15 languages, including low-resource languages such as Swahili and Urdu. We hope that our dataset, dubbed XNLI, will catalyze research in cross-lingual sentence understanding by providing an informative standard evaluation task. In addition, we provide several baselines for multilingual sentence understanding, including two based on machine translation systems, and two that use parallel data to train aligned multilingual bag-of-words and LSTM encoders. We find that XNLI represents a practical and challenging evaluation suite, and that directly translating the test data yields the best performance among available baselines.

Motivation & Objective

• Define a large-scale cross-lingual natural language inference (NLI) benchmark across 15 languages including low-resource ones.
• Evaluate translation-based baselines and multilingual sentence encoders for cross-language transfer in NLI.
• Propose and assess alignment-based multilingual sentence embeddings to transfer English NLI models to other languages.

Proposed method

• Extend English NLI data to 15 languages via professional translation of premises and hypotheses.
• Evaluate translation-based baselines: translate train and translate test.
• Evaluate multilingual sentence encoders: x-cbow (CBOW), x-bilstm (BiLSTM) with alignment loss.
• Propose an alignment loss L_align to align English and target language embeddings using parallel data.
• Compare against baselines using English-trained classifiers with multilingual encoders.
• Use parallel corpora (e.g., UN, Europarl, OpenSubtitles, IIT Bombay) to train alignment.

Experimental results

Research questions

• RQ1 How well do translation-based approaches perform for cross-lingual NLI across 15 languages?
• RQ2 Can multilingual sentence encoders aligned via a simple loss transfer NLI from English to other languages without translation during inference?
• RQ3 What is the impact of alignment loss and negative sampling on cross-lingual transfer performance?
• RQ4 How do low-resource languages (Urdu, Swahili) fare under translation-based vs alignment-based cross-lingual NLI?
• RQ5 What are the practical trade-offs between translation-based baselines and multilingual encoders for deployment?

Key findings

• Translation test baselines yield the best cross-lingual performance among baselines.
• XNLI transfer with multilingual sentence encoders is competitive with translate-train baselines but generally below translate-test performance by up to several points depending on language.
• BiLSTM encoders (BiLSTM-max) outperform CBOW baselines across languages.
• Alignment-based multilingual embeddings show promise and correlate with improved XNLI accuracy as alignment loss improves.
• For Urdu and Swahili, limited parallel data constrains gains in alignment-based methods, highlighting resource effects.