[Paper Review] The Missing Ingredient in Zero-Shot Neural Machine Translation

Naveen Arivazhagan, Ankur Bapna+4 more|arXiv (Cornell University)|Mar 17, 2019

Natural Language Processing Techniques39 references92 citations

TL;DR

The paper identifies that pure parameter sharing in multilingual NMT prevents effective zero-shot translation and proposes encoder-level auxiliary losses to enforce language-invariant representations, achieving zero-shot performance on par with pivoting on WMT14 and scalable results on IWSLT17.

ABSTRACT

Multilingual Neural Machine Translation (NMT) models are capable of translating between multiple source and target languages. Despite various approaches to train such models, they have difficulty with zero-shot translation: translating between language pairs that were not together seen during training. In this paper we first diagnose why state-of-the-art multilingual NMT models that rely purely on parameter sharing, fail to generalize to unseen language pairs. We then propose auxiliary losses on the NMT encoder that impose representational invariance across languages. Our simple approach vastly improves zero-shot translation quality without regressing on supervised directions. For the first time, on WMT14 English-FrenchGerman, we achieve zero-shot performance that is on par with pivoting. We also demonstrate the easy scalability of our approach to multiple languages on the IWSLT 2017 shared task.

Motivation & Objective

• Motivate the study by diagnosing why zero-shot translation under multilingual NMT with full parameter sharing underperforms.
• Propose and evaluate auxiliary losses to enforce language-invariant encoder representations.
• Demonstrate zero-shot performance on benchmark datasets reaches parity with pivoting without multi-phase data synthesis.
• Show scalability of the approach to additional languages (IWSLT17).
• Clarify the role of cross-lingual transfer as the missing ingredient for better zero-shot MT.

Proposed method

• Formulate zero-shot translation as a domain adaptation problem with English as the source domain and other languages as the target domain.
• Introduce an alignment regularization loss added to the standard cross-entropy translation loss (Loss = CE + lambda * Omega).
• Provide two classes of regularizers: (a) distribution-level alignment via a domain adversarial loss, (b) instance-level alignment leveraging known translation pairs (parallels) with a cosine similarity objective.
• Use a shared encoder/decoder Transformer base model with language tokens indicating target language; train end-to-end.
• Tune lambda to balance translation and alignment losses (lambda = 1.0 in experiments).
• Compare cosine-based alignment and adversarial alignment in terms of robustness and performance; highlight simplicity of cosine loss.

Experimental results

Research questions

• RQ1 Why do multilingual NMT models with weight tying fail to generalize to zero-shot language pairs?
• RQ2 Can explicit alignment of encoder representations across languages improve zero-shot translation without harming supervised directions?
• RQ3 Is English a practical pivot in latent space for cross-lingual transfer in multilingual NMT?
• RQ4 How do simple alignment losses compare to adversarial methods in improving zero-shot MT, and how scalable is the approach to more languages?

Key findings

• Baseline multilingual NMT exhibits strong supervised performance but weak zero-shot performance compared to pivoting, with notable language-entanglement and errors where outputs are translated into the wrong language.
• Zero-shot BLEU for de->fr improved from 17.00 (vanilla) to 26.00 (adversarial) and 25.85 (pool-cosine); fr->de improved from 11.84 (vanilla) to 20.39 (adversarial) and 20.18 (pool-cosine).
• Pivoting through English yielded BLEU scores of 26.25 (de->fr) and 20.18 (fr->de), illustrating that aligned latent representations can close the gap to pivoting.
• Cosine-based alignment is a simple, robust approach nearly matching the adversarial method, with fewer hyperparameters and less sensitivity.
• On IWSLT17, the cosine-aligned multilingual model matches pivoting and shows scalability to more languages (8–20 language configurations) with zero-shot results approaching or matching pivot performance.
• The alignment losses effectively disentangle source-language representations from target-language signals, enabling better zero-shot generalization without degrading supervised directions.