[Paper Review] Domain Robustness in Neural Machine Translation

Mathias Müller, Annette Rios+1 more|arXiv (Cornell University)|Nov 8, 2019

Natural Language Processing Techniques37 references63 citations

TL;DR

The paper analyzes domain robustness in SMT vs NMT, showing NMT is fluent but often inadequate in unseen domains due to hallucinations; it evaluates methods to improve robustness with moderate gains and provides baselines for future work.

ABSTRACT

Translating text that diverges from the training domain is a key challenge for machine translation. Domain robustness---the generalization of models to unseen test domains---is low for both statistical (SMT) and neural machine translation (NMT). In this paper, we study the performance of SMT and NMT models on out-of-domain test sets. We find that in unknown domains, SMT and NMT suffer from very different problems: SMT systems are mostly adequate but not fluent, while NMT systems are mostly fluent, but not adequate. For NMT, we identify such hallucinations (translations that are fluent but unrelated to the source) as a key reason for low domain robustness. To mitigate this problem, we empirically compare methods that are reported to improve adequacy or in-domain robustness in terms of their effectiveness at improving domain robustness. In experiments on German to English OPUS data, and German to Romansh (a low-resource setting) we find that several methods improve domain robustness. While those methods do lead to higher BLEU scores overall, they only slightly increase the adequacy of translations compared to SMT.

Motivation & Objective

• Assess how SMT and NMT perform on out-of-domain test sets.
• Characterize the nature of errors in out-of-domain translations (fluency vs adequacy).
• Evaluate methods that aim to improve domain robustness (adequacy, robustness, or out-of-domain translation).
• Provide datasets and code as baselines for future research on domain robustness.

Proposed method

• Compare baseline SMT and NMT systems on German→English and German→Romansh with multiple domains from OPUS/other corpora.
• Analyze automatically (BLEU, OOV rates) and manually (adequacy and fluency) to characterize domain robustness.
• Evaluate several robustness-improving techniques: subword regularization, defensive distillation, reconstruction, neural noisy channel reranking, and combinations thereof.
• Tune and test these methods on in-domain and out-of-domain data to assess cross-domain gains.

Experimental results

Research questions

• RQ1 How do SMT and NMT differ in degradation when translating out-of-domain text (fluency vs adequacy)?
• RQ2 Can techniques designed to improve adequacy or in-domain robustness also improve domain robustness across unseen domains?
• RQ3 Which methods most effectively reduce hallucinations and improve cross-domain translation quality?
• RQ4 Do improvements in BLEU translate to corresponding gains in adequacy across domains?
• RQ5 Are there domain- and data-condition dependent effects when combining robustness techniques?

Key findings

• NMT is fluent but often inadequate in out-of-domain translations, while SMT is more adequate but less fluent.
• Hallucinations (fluency without content relevance) are a key issue in NMT for out-of-domain data; SMT shows fewer hallucinations.
• Reconstruction and noisy channel reranking yield robust cross-domain gains and reduce hallucinations, with reconstruction particularly reducing inadequate translations.
• Subword regularization improves in low-resource in-domain and some out-of-domain settings but does not consistently boost cross-domain adequacy or BLEU.
• Combining reconstruction with noisy channel reranking provides the most consistent cross-domain BLEU improvement across the tested domains and data conditions.
• Overall, domain robustness remains challenging for NMT, and improvements in BLEU do not uniformly translate to adequacy gains across domains.