[Paper Review] Multilingual Denoising Pre-training for Neural Machine Translation

Yinhan Liu, Jiatao Gu+6 more|arXiv (Cornell University)|Jan 22, 2020

Natural Language Processing Techniques54 references607 citations

TL;DR

The paper introduces mBART, a multilingual sequence-to-sequence denoising auto-encoder pre-trained on 25 languages, which can be fine-tuned for supervised and unsupervised MT across many language pairs, yielding substantial gains especially in low/medium resource settings and enabling cross-language transfer.

ABSTRACT

This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART -- a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective. mBART is one of the first methods for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text. Pre-training a complete model allows it to be directly fine tuned for supervised (both sentence-level and document-level) and unsupervised machine translation, with no task-specific modifications. We demonstrate that adding mBART initialization produces performance gains in all but the highest-resource settings, including up to 12 BLEU points for low resource MT and over 5 BLEU points for many document-level and unsupervised models. We also show it also enables new types of transfer to language pairs with no bi-text or that were not in the pre-training corpus, and present extensive analysis of which factors contribute the most to effective pre-training.

Motivation & Objective

• Motivate and demonstrate that pre-training a complete multilingual Seq2Seq model with a denoising objective improves MT across diverse language pairs.
• Show that mBART pre-training yields significant gains for supervised sentence- and document-level MT, as well as unsupervised MT.
• Investigate how factors such as the number of languages, language similarity, and pre-training steps affect gains.
• Demonstrate transfer to language pairs with no bi-text and to languages not present in pre-training data, indicating language-universal benefits of the pre-trained model.

Proposed method

• Pre-train a multilingual sequence-to-sequence model (mBART) using a BART-style denoising objective on large-scale monolingual corpora across 25 languages (CC25).
• Use two noise types: span masking (35% of words) and sentence permutation, with a language id token to guide decoding.
• Train a single Transformer-based model (12-layer encoder and 12-layer decoder, model dim 1024, ~680M parameters) on multilingual data with 256 V100 GPUs for 500K steps.
• Fine-tune the pre-trained model on bilingual parallel data for supervised MT, using teacher forcing and standard decoding (beam size 5).
• Evaluate on sentence-level MT, document-level MT, and unsupervised MT tasks, including back-translation and language transfer scenarios.
• Compare to other pre-training approaches to establish the benefits of full Seq2Seq multilingual pre-training.

Experimental results

Research questions

• RQ1 Can a complete multilingual Seq2Seq model pre-trained with a denoising objective improve MT across low/medium/high-resource language pairs?
• RQ2 Does multilingual pre-training enable transfer to language pairs with no bi-text and to languages not present in pre-training data?
• RQ3 How do factors like number of pre-training languages, language similarity, and pre-training steps affect MT gains?
• RQ4 How does pre-training interact with back-translation and document-level MT tasks?
• RQ5 Is pre-training beneficial for unsupervised MT, including language transfer scenarios?

Key findings

• mBART pre-training yields gains across low and medium-resource bi-text pairs, including up to 12 BLEU points improvements.
• For high-resource pairs, pre-training shows less consistent gains and can slightly hurt when extensive bi-text is available.
• Back-translation combined with mBART initialization achieves new state-of-the-art results for several directions (e.g., En–Si, En–Ne in FLoRes).
• mBART outperforms other pre-training approaches (e.g., XLM, MASS, BART variants) on multiple En–Ro and related directions, especially with BT.
• Document-level MT benefits substantially from pre-training, with Doc-MT outperforming Sent-MT when pre-trained, and pre-training significantly outperforming non-pre-trained baselines.
• Unsupervised MT benefits from mBART pre-training via back-translation and language transfer, including strong performance on dissimilar pairs (e.g., En–Ne, En–Si) where prior methods struggle.