[Paper Review] A Brief Survey of Multilingual Neural Machine Translation

Raj Dabre, Chenhui Chu+1 more|arXiv (Cornell University)|May 14, 2019

Natural Language Processing Techniques96 references4 citations

TL;DR

This survey provides a comprehensive overview of multilingual neural machine translation (MNMT), categorizing approaches by resource scenarios and modeling principles. It highlights knowledge transfer, model efficiency, and cross-lingual representation learning as key benefits, offering a systematic framework for researchers to navigate MNMT architectures, datasets, and future directions such as zero-shot translation and multilingual representation learning.

ABSTRACT

We present a survey on multilingual neural machine translation (MNMT), which has gained a lot of traction in the recent years. MNMT has been useful in improving translation quality as a result of knowledge transfer. MNMT is more promising and interesting than its statistical machine translation counterpart because end-to-end modeling and distributed representations open new avenues. Many approaches have been proposed in order to exploit multilingual parallel corpora for improving translation quality. However, the lack of a comprehensive survey makes it difficult to determine which approaches are promising and hence deserve further exploration. In this paper, we present an in-depth survey of existing literature on MNMT. We categorize various approaches based on the resource scenarios as well as underlying modeling principles. We hope this paper will serve as a starting point for researchers and engineers interested in MNMT.

Motivation & Objective

• To systematize the rapidly growing body of research in multilingual neural machine translation (MNMT) by categorizing approaches based on resource availability and modeling principles.
• To identify and analyze key design choices, strengths, and limitations of existing MNMT systems across multiway, low/zero-resource, and multi-source translation scenarios.
• To provide a historical perspective on MNMT in relation to earlier multilingual statistical and rule-based MT systems.
• To compare MNMT with domain adaptation in NMT, highlighting similarities and differences in techniques such as transfer learning and pseudo-data generation.
• To outline promising future research directions, including language-agnostic representations, multi-target language modeling, and code-mixed language handling.

Proposed method

• Categorizing MNMT approaches into three main scenarios: multiway translation, low/zero-resource translation (including transfer learning, pivoting, and zero-shot translation), and multi-source translation.
• Analyzing modeling principles such as shared encoders, shared attention mechanisms, and adapter-based architectures for parameter efficiency.
• Surveying key datasets and shared tasks in multilingual MT, including OPUS, Tatoeba, and WMT benchmarks.
• Comparing MNMT with domain adaptation in NMT, particularly in techniques like back-translation for pseudo-parallel data and transfer learning.
• Evaluating the role of pre-training on cross-lingual word and sentence embeddings to improve generalization and zero-shot performance.
• Proposing future methodological directions, including language-agnostic representation learning and joint multilingual-multi-domain modeling.

Experimental results

Research questions

• RQ1 How can multilingual neural machine translation improve translation quality through knowledge transfer across diverse language pairs?
• RQ2 What are the most effective architectural designs for handling multiple languages in a single NMT model, especially in low-resource settings?
• RQ3 In what ways do multiway, zero-shot, and pivot-based translation approaches differ in performance and applicability?
• RQ4 How can pre-training on monolingual or parallel corpora enhance multilingual representation learning in NMT systems?
• RQ5 What are the challenges and opportunities in extending MNMT to handle code-mixed languages and multiple target languages simultaneously?

Key findings

• MNMT enables knowledge transfer across languages, significantly improving translation quality—especially for low-resource language pairs—by leveraging shared representations.
• Multiway NMT systems achieve compact model sizes by sharing parameters across multiple language pairs, reducing deployment costs on resource-constrained devices.
• Zero-shot translation is feasible in MNMT due to shared cross-lingual representations, allowing translation between language pairs without any parallel training data.
• Transfer learning and back-translation techniques in MNMT show strong performance gains, particularly when transferring knowledge from high-resource to low-resource languages.
• Cross-lingual sentence embeddings and pre-training methods such as mBERT and XLM-R have shown promise in improving zero-shot and few-shot translation performance.
• Despite progress, challenges remain in modeling language-agnostic representations, especially due to word-order divergence and variable-length representations across languages.