[Paper Review] A Survey of Deep Learning Techniques for Neural Machine Translation

Shuoheng Yang, Yuxin Wang+1 more|arXiv (Cornell University)|Feb 18, 2020

Natural Language Processing Techniques133 references98 citations

TL;DR

This paper surveys the origin and development of Neural Machine Translation (NMT), categorizing models by architecture and training approaches, highlighting attention and coverage mechanisms, and outlining future research trends.

ABSTRACT

In recent years, natural language processing (NLP) has got great development with deep learning techniques. In the sub-field of machine translation, a new approach named Neural Machine Translation (NMT) has emerged and got massive attention from both academia and industry. However, with a significant number of researches proposed in the past several years, there is little work in investigating the development process of this new technology trend. This literature survey traces back the origin and principal development timeline of NMT, investigates the important branches, categorizes different research orientations, and discusses some future research trends in this field.

Motivation & Objective

• Trace the origin and development timeline of Neural Machine Translation (NMT).
• Categorize NMT models by architectural orientation and training approaches.
• Discuss key components (attention, vocabulary coverage) and their impact on translation quality.
• Evaluate advanced NMT models and potential future directions.

Proposed method

• Literature review of historical MT paradigms (rule-based, statistical, neural).
• Classification of NMT variants by architecture (RNN-based, CNN-based, Transformer).
• Discussion of encoder–decoder structure and training/inference procedures.
• Analysis of attention mechanisms and coverage problems in NMT.
• Overview of advanced models and practical implications for speed and quality.

Experimental results

Research questions

• RQ1 What are the major development stages and model families in NMT history?
• RQ2 How do attention and coverage mechanisms improve translation quality in NMT?
• RQ3 What are the trade-offs between RNN, CNN, and Transformer-based NMT models?
• RQ4 What are the highlighted future trends and directions in NMT research?

Key findings

• NMT evolved from shallow to deep architectures and to attention-based and Transformer models.
• Encoder–decoder is the core structure for end-to-end NMT and is enhanced via depth, bidirectionality, and specialized units (LSTM/GRU).
• Attention mechanisms address long-range dependencies and improve translation quality beyond early RNN/NLM baselines.
• CNN-based NMT offers speed advantages but historically struggled with long-range dependencies until attention/Transformer refinements.
• Vocabulary coverage and related mechanisms are essential considerations in current NMT designs.
• The survey discusses advanced models (e.g., GNMT, MoE) and anticipates future research directions in NMT.