[Paper Review] Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder

Thanh-Le Ha, Jan Niehues+1 more|arXiv (Cornell University)|Nov 15, 2016

Natural Language Processing TechniquesComputer Science15 references266 citations

TL;DR

The paper presents a unified multilingual NMT framework with a single shared encoder and decoder, using language-specific coding and target forcing to enable many-to-many translation without architecture changes, showing improvements in under-resourced and zero-resource settings.

ABSTRACT

In this paper, we present our first attempts in building a multilingual Neural Machine Translation framework under a unified approach. We are then able to employ attention-based NMT for many-to-many multilingual translation tasks. Our approach does not require any special treatment on the network architecture and it allows us to learn minimal number of free parameters in a standard way of training. Our approach has shown its effectiveness in an under-resourced translation scenario with considerable improvements up to 2.6 BLEU points. In addition, the approach has achieved interesting and promising results when applied in the translation task that there is no direct parallel corpus between source and target languages.

Motivation & Objective

• Motivate and develop a multilingual NMT framework that uses a single encoder and a single decoder for multiple languages.
• Avoid architectural changes while enabling attention mechanisms in multilingual translation.
• Leverage language-specific coding and target forcing to control translation direction and language alignment.
• Evaluate the approach on under-resourced and zero-resource translation scenarios to assess gains and limitations.

Proposed method

• Adopt a single universal encoder and decoder across languages without modifying the attention mechanism.
• Apply language-specific coding by prefixing words with language codes (e.g., @de@, @en@) to place different languages into shared embeddings.
• Introduce target forcing by prepending/appending a language-specific sentinel to source sentences to steer translation toward a chosen target language.
• Train with standard NMT training procedures (shared encoder/decoder; same attention) and use BPE subword units for vocabulary management.
• Use a mix-source or mix-multi-source strategy to incorporate monolingual data and auxiliary languages to improve low-resource translations.
• Evaluate with BLEU on under-resourced English-German and zero-resource German-French scenarios; compare against baselines and pivot/bridge variants.

Experimental results

Research questions

• RQ1 Can a single universal encoder and decoder handle many-to-many multilingual translation without architectural changes?
• RQ2 Does language-specific coding plus target forcing enable effective multilingual NMT and reduce parameter count?
• RQ3 How does multilingual training with mix-source and mix-multi-source strategies perform in under-resourced and zero-resource settings?
• RQ4 What is the impact of using large monolingual corpora within this unified multilingual framework?

Key findings

• In under-resourced English→German translation, Mix-source improves BLEU by 2.64 on tst2013 and 2.09 on tst2014 over the baseline; Mix-multi-source improves by 2.21 and 1.59 respectively.
• Incorporating large monolingual data with a mix-source setup yields a BLEU gain of 1.52 on tst2013 and 1.06 on tst2014 after adaptation on parallel data.
• Zero-resource German→French experiments show Bridge and Universal strategies underperform the Pivot baseline, with BLEU scores of 9.70 and 10.77 respectively, compared to Pivot at 13.41; language identification errors are non-negligible (Bridge ~21.27% words wrong language; Universal ~17.57%).
• The approach demonstrates multilinguality benefits and competitive results without architecture redesign, while highlighting balancing and forcing mechanism challenges in zero-resource scenarios.