[Paper Review] Multi-Source Neural Machine Translation with Missing Data

Yuta Nishimura, Katsuhito Sudoh+2 more|arXiv (Cornell University)|Jun 7, 2018

Natural Language Processing Techniques11 references4 citations

TL;DR

This paper proposes a simple yet effective method for multi-source neural machine translation (NMT) with incomplete multilingual corpora by replacing missing source translations with a special $<$NULL$>$ token. The approach enables both training and inference on incomplete data using standard multi-encoder and mixture-of-experts NMT architectures, achieving BLEU gains of up to 2.22 points over one-to-one NMT systems on real TED Talks data.

ABSTRACT

Multi-source translation is an approach to exploit multiple inputs (e.g. in two different languages) to increase translation accuracy. In this paper, we examine approaches for multi-source neural machine translation (NMT) using an incomplete multilingual corpus in which some translations are missing. In practice, many multilingual corpora are not complete due to the difficulty to provide translations in all of the relevant languages (for example, in TED talks, most English talks only have subtitles for a small portion of the languages that TED supports). Existing studies on multi-source translation did not explicitly handle such situations. This study focuses on the use of incomplete multilingual corpora in multi-encoder NMT and mixture of NMT experts and examines a very simple implementation where missing source translations are replaced by a special symbol . These methods allow us to use incomplete corpora both at training time and test time. In experiments with real incomplete multilingual corpora of TED Talks, the multi-source NMT with the tokens achieved higher translation accuracies measured by BLEU than those by any one-to-one NMT systems.

Motivation & Objective

• To address the challenge of training and deploying multi-source NMT systems when multilingual corpora are incomplete due to missing translations.
• To enable effective use of all available source language translations at both training and inference time, even when some languages are missing.
• To evaluate whether simple replacement of missing inputs with a $<$NULL$>$ token can improve translation performance compared to standard one-to-one NMT.
• To investigate the performance of multi-source NMT in real-world scenarios like multilingual subtitles, where full coverage is rare.

Proposed method

• Replace missing source sentences in multi-source NMT with a learned $<$NULL$>$ token during both training and inference.
• Use standard multi-encoder NMT architecture with global attention, where decoder hidden states are initialized via a weighted concatenation of encoder final states.
• Apply the same $<$NULL$>$ token strategy in mixture-of-NMT-experts models, where each expert is trained on a bilingual subset and outputs are gated by a routing network.
• Train and evaluate models on real incomplete multilingual corpora from TED Talks, including cases with missing translations in French, Portuguese, or other languages.
• Use BLEU score as the primary metric to compare multi-source NMT with $<$NULL$>$ handling against one-to-one NMT systems.
• Evaluate performance on both complete (no missing data) and incomplete (partial data) test sets to assess robustness.

Experimental results

Research questions

• RQ1 Can multi-source NMT systems achieve higher translation accuracy when trained and tested on incomplete multilingual corpora using a $<$NULL$>$ token to represent missing inputs?
• RQ2 How does the performance of multi-source NMT with $<$NULL$>$ tokens compare to one-to-one NMT systems in terms of BLEU score on real-world incomplete data?
• RQ3 Does the presence of even one additional source language (e.g., French) improve translation quality when the other source (e.g., Brazilian Portuguese) is missing?
• RQ4 How do different multi-source NMT architectures—multi-encoder and mixture-of-experts—behave under missing data conditions?

Key findings

• The multi-source NMT model with $<$NULL$>$ tokens achieved a BLEU score gain of 2.22 points over one-to-one NMT in the {En,Es,Fr}-to-Pt (br) translation task.
• In the {En,Fr,Pt (br)}-to-Es task, the multi-encoder model improved BLEU by 1.69 points, and the mixture-of-experts model by 1.19 points.
• For the {En,Es,Pt (br)}-to-Fr task, the mixture-of-experts model achieved a 1.69-point BLEU improvement over one-to-one NMT.
• Even when only one source sentence was available (e.g., English only), the multi-source models produced translations identical to the reference, demonstrating robustness to missing data.
• The mixture-of-NMT-experts model achieved a BLEU+1 score of 0.726 in Example (1), significantly outperforming the one-to-one model (0.266), despite missing Brazilian Portuguese input.
• All performance improvements were statistically significant (p < 0.01), confirming the effectiveness of the $<$NULL$>$ approach across multiple language combinations and model architectures.