[Paper Review] Multilingual Translation with Extensible Multilingual Pretraining and Finetuning

Yuqing Tang, Chau Tran+6 more|arXiv (Cornell University)|Aug 2, 2020

Topic Modeling24 references151 citations

TL;DR

The paper shows that multilingual translation models can be built by multilingual finetuning of pretrained models (like mBART), extendable to 50 languages (mBART50) without retraining from scratch, and introduces the ML50 benchmark for reproducible multilingual MT research.

ABSTRACT

Recent work demonstrates the potential of multilingual pretraining of creating one model that can be used for various tasks in different languages. Previous work in multilingual pretraining has demonstrated that machine translation systems can be created by finetuning on bitext. In this work, we show that multilingual translation models can be created through multilingual finetuning. Instead of finetuning on one direction, a pretrained model is finetuned on many directions at the same time. Compared to multilingual models trained from scratch, starting from pretrained models incorporates the benefits of large quantities of unlabeled monolingual data, which is particularly important for low resource languages where bitext is not available. We demonstrate that pretrained models can be extended to incorporate additional languages without loss of performance. We double the number of languages in mBART to support multilingual machine translation models of 50 languages. Finally, we create the ML50 benchmark, covering low, mid, and high resource languages, to facilitate reproducible research by standardizing training and evaluation data. On ML50, we demonstrate that multilingual finetuning improves on average 1 BLEU over the strongest baselines (being either multilingual from scratch or bilingual finetuning) while improving 9.3 BLEU on average over bilingual baselines from scratch.

Motivation & Objective

• Demonstrate that multilingual finetuning of pretrained models yields strong multilingual translation performance.
• Show that extending pretrained models to include additional languages does not harm original language performance.
• Introduce a standardized benchmark (ML50) spanning high, mid, and low resource languages for reproducible multilingual MT research.

Proposed method

• Use mBART as the base pretrained multilingual denoising autoencoder.
• Fine-tune the pretrained model on multilingual bitext across many language directions (N languages to N languages).
• Add language tokens to input/output to specify source/target languages during fine-tuning.
• Explore three model variants: Many-to-one, one-to-many, and many-to-many (via English pivot).
• Apply temperature-based upsampling to balance multilingual training data across language pairs.

Experimental results

Research questions

• RQ1 Can multilingual finetuning of a pretrained model outperform bilingual finetuning and multilingual training from scratch?
• RQ2 Can an existing pretrained model (mBART) be extended to include additional languages without retraining from scratch or degrading performance on original languages?
• RQ3 Does multilingual finetuning provide consistent gains across Many-to-one, one-to-many, and many-to-many translation directions?
• RQ4 What is the impact of adding languages on a pretrained model’s original language performance?
• RQ5 How does the proposed ML50 benchmark support reproducible evaluation in multilingual MT?

Key findings

• Multilingual finetuning yields average BLEU gains over bilingual baselines, notably +3.6 to English in Many-to-English and +3.0 BLEU in certain settings for English-to-Many configurations (as reported in abstracted results).
• Extending mBART to 50 languages (mBART50) does not degrade performance on the original 25 languages when fine-tuned for translation tasks.
• In the 50-language setting, ML-FT achieves average improvements over baselines of 2.61 BLEU (Many-to-one), -0.47 BLEU (one-to-Many), and -0.15 to -0.35 BLEU (Many-to-Many) compared to strongest baselines, indicating robust performance gains especially for low-resource pairs.
• ML50 benchmark standardizes data sources, languages, and evaluation to enable reproducible multilingual MT research (230M bitext; 50 languages across varied resource levels).
• Compared to bilingual finetuning, multilingual finetuning can yield up to 12.3 BLEU average improvements into English across 50 directions, with some low-resource languages showing 10+ BLEU gains in Many-to-one translations.
• Overall, multilingual finetuning consistently outperforms the strongest baselines on average when combining Many-to-one and Many-to-Many directions.