[Paper Review] Exploiting Out-of-Domain Parallel Data through Multilingual Transfer Learning for Low-Resource Neural Machine Translation

Aizhan Imankulova, Raj Dabre+2 more|arXiv (Cornell University)|Jul 6, 2019

Natural Language Processing Techniques26 references18 citations

TL;DR

This paper proposes a multilingual multistage fine-tuning approach that leverages out-of-domain parallel data to improve low-resource Japanese--Russian neural machine translation. By first pre-training a multilingual model on diverse out-of-domain data and then fine-tuning it on in-domain parallel and back-translated data, the method achieves a +3.7 BLEU improvement over a strong baseline in an extremely low-resource setting.

ABSTRACT

This paper proposes a novel multilingual multistage fine-tuning approach for low-resource neural machine translation (NMT), taking a challenging Japanese--Russian pair for benchmarking. Although there are many solutions for low-resource scenarios, such as multilingual NMT and back-translation, we have empirically confirmed their limited success when restricted to in-domain data. We therefore propose to exploit out-of-domain data through transfer learning, by using it to first train a multilingual NMT model followed by multistage fine-tuning on in-domain parallel and back-translated pseudo-parallel data. Our approach, which combines domain adaptation, multilingualism, and back-translation, helps improve the translation quality by more than 3.7 BLEU points, over a strong baseline, for this extremely low-resource scenario.

Motivation & Objective

• To address the persistent challenge of low-resource neural machine translation, especially for low-resource language pairs like Japanese--Russian.
• To investigate whether out-of-domain parallel data can be effectively leveraged in low-resource NMT scenarios where in-domain data is scarce.
• To improve translation quality by combining multilingual pre-training, domain adaptation, and back-translation in a multistage fine-tuning framework.
• To demonstrate that transfer learning from diverse, out-of-domain parallel data can significantly enhance performance in extremely low-resource settings.

Proposed method

• Pre-train a multilingual NMT model on large-scale out-of-domain parallel data across multiple languages.
• Fine-tune the multilingual model on in-domain parallel monolingual data specific to the Japanese--Russian translation task.
• Apply back-translation to generate pseudo-parallel monolingual data from in-domain monolingual sentences in the target language.
• Perform multistage fine-tuning by successively adapting the model to in-domain parallel data and then to back-translated pseudo-parallel data.
• Use a shared multilingual encoder-decoder architecture to maintain cross-lingual transferability throughout the training process.

Experimental results

Research questions

• RQ1 Can out-of-domain parallel data improve low-resource neural machine translation when in-domain data is extremely limited?
• RQ2 How does combining multilingual pre-training with domain-specific fine-tuning affect translation quality in low-resource settings?
• RQ3 What is the contribution of back-translation when integrated into a multilingual multistage fine-tuning pipeline?
• RQ4 Does the proposed method outperform strong baselines that rely solely on in-domain data or standard multilingual NMT?

Key findings

• The proposed method achieves a +3.7 BLEU point improvement over a strong baseline in the low-resource Japanese--Russian translation task.
• The use of out-of-domain parallel data significantly enhances model performance, even when the data is not directly aligned with the target domain.
• Multistage fine-tuning on in-domain parallel and back-translated data leads to better domain adaptation than single-stage fine-tuning.
• The combination of multilingualism, domain adaptation, and back-translation yields superior results compared to standard approaches in low-resource settings.