[Paper Review] Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation

Aditya Siddhant, Ankur Bapna+6 more|arXiv (Cornell University)|May 11, 2020

Natural Language Processing TechniquesComputer Science24 references35 citations

TL;DR

The paper integrates monolingual data with self-supervised learning into multilingual NMT to boost low-resource language translation and enable effective addition of unseen languages without parallel data.

ABSTRACT

Over the last few years two promising research directions in low-resource neural machine translation (NMT) have emerged. The first focuses on utilizing high-resource languages to improve the quality of low-resource languages via multilingual NMT. The second direction employs monolingual data with self-supervision to pre-train translation models, followed by fine-tuning on small amounts of supervised data. In this work, we join these two lines of research and demonstrate the efficacy of monolingual data with self-supervision in multilingual NMT. We offer three major results: (i) Using monolingual data significantly boosts the translation quality of low-resource languages in multilingual models. (ii) Self-supervision improves zero-shot translation quality in multilingual models. (iii) Leveraging monolingual data with self-supervision provides a viable path towards adding new languages to multilingual models, getting up to 33 BLEU on ro-en translation without any parallel data or back-translation.

Motivation & Objective

• Motivate combining multilingual transfer learning with self-supervised monolingual pre-training for NMT.
• Demonstrate that monolingual data improves translation quality across languages in a multilingual setup.
• Show that self-supervision enhances zero-shot translation in multilingual models.
• Illustrate a path to add new languages to multilingual NMT using only monolingual data.
• Provide empirical evidence on WMT data across multiple language directions.

Proposed method

• Use MASS-based masked sequence-to-sequence pre-training adapted to multilingual NMT.
• Prepend a target language token <2xx> to source sentences to denote the desired output language.
• Train jointly on supervised parallel data (translation objective) and monolingual data with the MASS objective.
• Balance language data with temperature-based sampling to address resource imbalances.
• Mix monolingual self-supervision and parallel data in a 50/50 ratio when both sources are used.
• Evaluate with SacreBLEU on standard WMT validation/test sets.

Experimental results

Research questions

• RQ1 Does incorporating monolingual data with self-supervision improve translation quality for low-resource languages in a multilingual NMT setting?
• RQ2 Does self-supervision improve zero-shot translation quality in multilingual models?
• RQ3 Can monolingual data with self-supervision help add new unseen languages to a multilingual model without parallel data?
• RQ4 How does the approach compare to bilingual baselines and pivot-based or other multilingual setups across language directions?

Key findings

• Adding monolingual data yields significant quality gains for low-resource languages within multilingual NMT.
• Self-supervision improves zero-shot translation performance, approaching pivot-based methods without alignment or adversarial losses.
• Monolingual data with self-supervision enables adding new languages to a multilingual model with strong translation quality, often close to fully supervised multilingual baselines.
• In high-resource directions, multilingual models with monolingual data match or closely approach bilingual baselines (with some exceptions).
• For certain very low-resource languages, gains from monolingual data are particularly large (3–5 BLEU, and higher in some directions).
• On ro-en, the model with monolingual data outperforms XLM and matches MASS-based bilingual performance without back-translation.