[Paper Review] Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation

Biao Zhang, Philip Williams+2 more|arXiv (Cornell University)|Apr 24, 2020

Natural Language Processing TechniquesComputer Science35 references82 citations

TL;DR

The paper strengthens massively multilingual NMT with language-aware components, deeper architectures, and Random Online Backtranslation (ROBt), achieving closer performance to bilingual models and pivot-based methods, and substantially boosting zero-shot translation.

ABSTRACT

Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. In this paper, we explore ways to improve them. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics, and overcome this bottleneck via language-specific components and deepening NMT architectures. We identify the off-target translation issue (i.e. translating into a wrong target language) as the major source of the inferior zero-shot performance, and propose random online backtranslation to enforce the translation of unseen training language pairs. Experiments on OPUS-100 (a novel multilingual dataset with 100 languages) show that our approach substantially narrows the performance gap with bilingual models in both one-to-many and many-to-many settings, and improves zero-shot performance by ~10 BLEU, approaching conventional pivot-based methods.

Motivation & Objective

• Motivate and address capacity bottlenecks in massively multilingual NMT when handling diverse languages.
• Improve zero-shot translation quality and reduce off-target translations.
• Explore architecture and data-driven strategies to boost multilingual translation performance.
• Evaluate the effectiveness of language-aware components and deep Transformer architectures.
• Propose and assess a scalable backtranslation-based finetuning method for zero-shot directions.

Proposed method

• Adopt Transformer-based multilingual NMT with English-centric data (OPUS-100).
• Introduce language-aware layer normalization (LaLn) to condition normalization on the target language token.
• Introduce language-aware linear transformation (LaLt) between encoder and decoder to adapt translation mappings per target language.
• Deepen Transformer architecture to increase modeling capacity.
• Develop Random Online Backtranslation (ROBt): online backtranslation with randomly sampled intermediate languages to generate pseudo-parallel data for zero-shot directions during finetuning.
• Evaluate both one-to-many and many-to-many settings on OPUS-100 and report BLEU and translation-language accuracy for zero-shot translations.

Experimental results

Research questions

• RQ1 How does increasing modeling capacity affect multilingual NMT performance across many-to-many language directions?
• RQ2 Can language-aware normalization and language-aware linear transforms alleviate capacity bottlenecks and improve zero-shot translation?
• RQ3 Does online backtranslation (ROBt) reduce off-target translations and improve zero-shot BLEU, approaching pivot-based methods?
• RQ4 How do deeper Transformer architectures compare to language-aware components in terms of performance and scalability for massively multilingual NMT?
• RQ5 What is the impact of training data size across language pairs on the effectiveness of these methods?

Key findings

• Multilingual NMT gains large improvements with increased model capacity, narrowing the gap to bilingual models, especially for low-resource languages.
• Language-aware modeling (LaLn and LaLt) substantially improves zero-shot performance and reduces off-target translations, with LaLt delivering notable gains.
• Deepening the Transformer yields benefits and, when combined with LaLn and LaLt, yields the best results among the ablations.
• Random Online Backtranslation (ROBt) reduces off-target translations by about 50% and increases zero-shot BLEU by roughly 10 points, approaching pivot-based methods, and converges within a few thousand steps.
• On OPUS-100 (55M sentence pairs across 100 languages), the combination of deep Transformer, LaLn, LaLt, and ROBt narrows the gap with bilingual NMT and pivot methods, while achieving substantial zero-shot improvements.
• Zero-shot translation accuracy (ACC zero) improves from around 35–50% gains to about 85–87% with ROBt, and zero-shot BLEU gains of up to 10.11 are observed.