[Paper Review] Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks

Haoyang Huang, Yaobo Liang+5 more|arXiv (Cornell University)|Sep 3, 2019

Topic ModelingComputer Science25 references110 citations

TL;DR

Unicoder pre-trains a universal language encoder using five cross-lingual tasks and multi-language fine-tuning, achieving state-of-the-art results on XNLI and XQA compared to Multilingual BERT and XLM baselines.

ABSTRACT

We present Unicoder, a universal language encoder that is insensitive to different languages. Given an arbitrary NLP task, a model can be trained with Unicoder using training data in one language and directly applied to inputs of the same task in other languages. Comparing to similar efforts such as Multilingual BERT and XLM, three new cross-lingual pre-training tasks are proposed, including cross-lingual word recovery, cross-lingual paraphrase classification and cross-lingual masked language model. These tasks help Unicoder learn the mappings among different languages from more perspectives. We also find that doing fine-tuning on multiple languages together can bring further improvement. Experiments are performed on two tasks: cross-lingual natural language inference (XNLI) and cross-lingual question answering (XQA), where XLM is our baseline. On XNLI, 1.8% averaged accuracy improvement (on 15 languages) is obtained. On XQA, which is a new cross-lingual dataset built by us, 5.5% averaged accuracy improvement (on French and German) is obtained.

Motivation & Objective

• Motivate the need for cross-lingual transfer learning across languages with limited labeled data.
• Propose a universal encoder trained with multiple cross-lingual pre-training tasks to learn language-agnostic representations.
• Show that multi-language fine-tuning further improves cross-lingual transfer.
• Empirically evaluate on XNLI and XQA against strong multilingual baselines and establish new state-of-the-art results.

Proposed method

• Introduce Unicoder, a 12-layer transformer with 1024 hidden units and 95k vocabulary, shared across 15 languages.
• Pre-train with five tasks: masked language modeling (MLM), translation language modeling (TLM), cross-lingual word recovery, cross-lingual paraphrase classification, and cross-lingual masked language modeling (document-level).
• Use a bilingual sentence pair attention mechanism to recover original words in cross-lingual word recovery.
• Train cross-lingual paraphrase classifier by concatenating sentences from two languages and using the first token representation for binary classification.
• Construct cross-lingual documents by aligning sentences across languages and apply MLM on 256-token sequences.
• Fine-tune with a Multi-language Fine-tuning strategy, jointly training on data from multiple languages (real or pseudo translations) to improve transfer.

Experimental results

Research questions

• RQ1 Can a single universal encoder trained with diverse cross-lingual tasks perform well across languages with minimal or no language-specific adaptation?
• RQ2 Do additional cross-lingual pre-training tasks beyond MLM and TLM improve cross-lingual transfer performance?
• RQ3 Does multi-language fine-tuning further boost cross-lingual performance, and how does language count affect gains?
• RQ4 How does Unicoder compare to strong baselines (Multilingual BERT, XLM) on XNLI and XQA?
• RQ5 What are the effects of joint fine-tuning on English plus other languages for cross-lingual tasks?

Key findings

• Unicoder achieves state-of-the-art performance on XNLI, especially with Multi-language Fine-tuning, reaching 78.5% average accuracy in the best setup.
• On XNLI, Unicoder outperforms XLM across fine-tuning settings, with gains of up to 1.6% when combining with Multi-language Fine-tuning.
• On the new XQA dataset, Unicoder achieves 69.7% average accuracy (with English plus other languages fine-tuning and translate-train), surpassing the XLM baseline by up to 5.5 percentage points.
• Ablation shows removing any of the three new cross-lingual tasks reduces performance, with Word Recovery contributing notably to gains; paraphrase classification shows relatively smaller drops.
• Multi-language Fine-tuning consistently improves performance, and more languages in fine-tuning generally yield better results, though some language pairs show exceptions.