[Paper Review] Multilingual Neural Machine Translation for Zero-Resource Languages

Surafel M. Lakew, Marcello Federico+2 more|arXiv (Cornell University)|Sep 16, 2019

Natural Language Processing Techniques44 references4 citations

TL;DR

This paper proposes a multilingual neural machine translation (NMT) framework using shared semantic representations across English, Italian, and Romanian to improve zero-shot translation for low-resource language pairs. By iteratively self-training on monolingual data generated from the model's own translations, the approach achieves performance comparable to fully supervised bilingual systems, outperforming pivoting baselines and demonstrating effective parameter transfer in low-resource settings.

ABSTRACT

In recent years, Neural Machine Translation (NMT) has been shown to be more effective than phrase-based statistical methods, thus quickly becoming the state of the art in machine translation (MT). However, NMT systems are limited in translating low-resourced languages, due to the significant amount of parallel data that is required to learn useful mappings between languages. In this work, we show how the so-called multilingual NMT can help to tackle the challenges associated with low-resourced language translation. The underlying principle of multilingual NMT is to force the creation of hidden representations of words in a shared semantic space across multiple languages, thus enabling a positive parameter transfer across languages. Along this direction, we present multilingual translation experiments with three languages (English, Italian, Romanian) covering six translation directions, utilizing both recurrent neural networks and transformer (or self-attentive) neural networks. We then focus on the zero-shot translation problem, that is how to leverage multi-lingual data in order to learn translation directions that are not covered by the available training material. To this aim, we introduce our recently proposed iterative self-training method, which incrementally improves a multilingual NMT on a zero-shot direction by just relying on monolingual data. Our results on TED talks data show that multilingual NMT outperforms conventional bilingual NMT, that the transformer NMT outperforms recurrent NMT, and that zero-shot NMT outperforms conventional pivoting methods and even matches the performance of a fully-trained bilingual system.

Motivation & Objective

• Address the challenge of low-resource language translation where parallel training data is scarce or unavailable.
• Explore whether multilingual NMT can improve zero-shot translation between language pairs not present in the training data.
• Develop a self-training method that leverages monolingual data to iteratively improve zero-shot translation performance.
• Compare the effectiveness of recurrent NMT and Transformer architectures in multilingual and zero-shot settings.
• Demonstrate that self-training can close the performance gap between zero-shot and fully supervised translation systems.

Proposed method

• Train a single multilingual NMT model on parallel data for three languages (English, Italian, Romanian), using language-specific tokens to condition translation directions.
• Apply a zero-shot translation setup where the model translates between Italian and Romanian despite no direct parallel data for that pair.
• Implement an iterative self-training loop: generate monolingual translations from the model, filter and retrain on the best hypotheses, and repeat for multiple rounds.
• Use a dual-inference strategy to generate parallel-like data in both directions (e.g., Italian→Romanian and Romanian→Italian) simultaneously.
• Evaluate performance using BLEU scores on held-out test sets, comparing against bilingual baselines and pivoting methods.
• Fine-tune the model over 5 rounds of self-training, with each round involving up to 3 epochs on augmented data.

Experimental results

Research questions

• RQ1 Can multilingual NMT with shared representations improve zero-shot translation performance for low-resource language pairs?
• RQ2 How does the performance of multilingual NMT compare to conventional bilingual NMT and pivoting methods in zero-shot settings?
• RQ3 To what extent can iterative self-training using model-generated monolingual data improve zero-shot translation quality?
• RQ4 Does the Transformer architecture outperform recurrent NMT in multilingual and zero-shot translation scenarios?
• RQ5 Can self-training on model-generated data eventually match or exceed the performance of fully supervised bilingual systems?

Key findings

• The multilingual NMT model outperforms conventional bilingual NMT systems in low-resource settings, achieving higher BLEU scores with shared parameters across languages.
• The Transformer architecture consistently outperforms the recurrent NMT model across all translation directions, including zero-shot setups.
• The iterative self-training method improves zero-shot translation performance by +10.50 BLEU points (from 4.72 to 15.22) on Italian→Romanian and +11.22 BLEU points (from 5.09 to 16.31) on Romanian→Italian in the first round.
• After five rounds of self-training, the model achieves 20.11 BLEU on Italian→Romanian and 22.41 BLEU on Romanian→Italian, matching or slightly exceeding the performance of fully supervised bilingual systems (20.10 and 21.36 BLEU, respectively).
• The self-training approach outperforms both single-pair and multilingual pivoting methods, with a +3.34 and +3.02 BLEU gain over multilingual pivoting on the two zero-shot directions at round 5.
• The results demonstrate that self-training on model-generated data leads to cleaner, higher-quality translations over time, indicating effective self-correction and progressive learning.