[Paper Review] A Survey of Domain Adaptation for Neural Machine Translation

Chenhui Chu, Rui Wang|arXiv (Cornell University)|Jun 1, 2018

Natural Language Processing Techniques80 references130 citations

TL;DR

A comprehensive survey of domain adaptation techniques for neural machine translation (NMT), organized into data-centric and model-centric approaches, with discussions on real-world applicability and future directions.

ABSTRACT

Neural machine translation (NMT) is a deep learning based approach for machine translation, which yields the state-of-the-art translation performance in scenarios where large-scale parallel corpora are available. Although the high-quality and domain-specific translation is crucial in the real world, domain-specific corpora are usually scarce or nonexistent, and thus vanilla NMT performs poorly in such scenarios. Domain adaptation that leverages both out-of-domain parallel corpora as well as monolingual corpora for in-domain translation, is very important for domain-specific translation. In this paper, we give a comprehensive survey of the state-of-the-art domain adaptation techniques for NMT.

Motivation & Objective

• Motivate the need for domain-specific translation in scenarios where domain-specific corpora are scarce.
• Survey and categorize domain adaptation techniques for NMT, distinguishing data-centric and model-centric approaches.
• Summarize how in-domain monolingual data, synthetic data, and out-of-domain parallel data can be leveraged for NMT domain adaptation.
• Discuss practical considerations, real-world deployment challenges, and directions for future research in domain-adaptive NMT.

Proposed method

• Categorizes domain adaptation methods for NMT into data-centric and model-centric paradigms.
• Within data-centric: reviews use of monolingual data, synthetic parallel data generation, and out-of-domain parallel data with domain tags and oversampling.
• Within model-centric: covers training objective adjustments, architecture adaptations (deep fusion, domain discriminators, domain control), and decoding-centric strategies (shallow fusion, ensembling).
• Discusses multi-domain training, data selection, and mixed fine-tuning as practical strategies for combining domain data.
• Provides discussion of real-world scenarios and guidance on selecting methods based on data availability.

Experimental results

Research questions

• RQ1 What domain adaptation techniques are effective for neural machine translation in the presence of limited in-domain parallel data?
• RQ2 How can monolingual in-domain data and synthetic data be leveraged to improve in-domain translation quality in NMT?
• RQ3 What are the trade-offs between data-centric and model-centric domain adaptation strategies for NMT, and how do they transfer to real-world deployment?
• RQ4 How should domain information be incorporated (e.g., domain tags, discriminators) to control or improve domain-specific translation in NMT?
• RQ5 What are promising future directions for applying domain adaptation to state-of-the-art NMT architectures and multilingual settings?

Key findings

• The survey identifies two main categories for NMT domain adaptation: data-centric and model-centric approaches.
• Data-centric methods include leveraging in-domain monolingual data, generating synthetic parallel data via back-translation, and using out-of-domain data with domain tagging or data selection techniques.
• Model-centric methods include training objective adjustments, architecture changes (such as deep fusion and domain discriminators), and decoding strategies (like shallow fusion and lattice-based decoding).
• Multi-domain and data selection strategies are discussed, with mixed fine-tuning often providing practical benefits by preserving out-of-domain performance while improving in-domain translation.
• The paper highlights the need to adapt domain techniques to state-of-the-art NMT architectures (RNNs, CNNs, Transformer) and discusses future directions like adversarial domain adaptation, domain generation, and multilingual/multi-domain setups.