[Paper Review] Learning to Reuse Translations: Guiding Neural Machine Translation with Examples

Qian Cao, Shaohui Kuang+1 more|arXiv (Cornell University)|Nov 25, 2019

Natural Language Processing Techniques33 references4 citations

TL;DR

This paper proposes Example-Guided Neural Machine Translation (EGNMT), a framework that enhances NMT by reusing relevant translation segments from similar example sentences. It introduces two models: a noise-masked encoder to filter irrelevant words in examples and an auxiliary decoder to jointly train the primary decoder to predict reusable phrases, achieving up to +9 BLEU points over baseline systems on multiple language pairs.

ABSTRACT

In this paper, we study the problem of enabling neural machine translation (NMT) to reuse previous translations from similar examples in target prediction. Distinguishing reusable translations from noisy segments and learning to reuse them in NMT are non-trivial. To solve these challenges, we propose an Example-Guided NMT (EGNMT) framework with two models: (1) a noise-masked encoder model that masks out noisy words according to word alignments and encodes the noise-masked sentences with an additional example encoder and (2) an auxiliary decoder model that predicts reusable words via an auxiliary decoder sharing parameters with the primary decoder. We define and implement the two models with the state-of-the-art Transformer. Experiments show that the noise-masked encoder model allows NMT to learn useful information from examples with low fuzzy match scores (FMS) while the auxiliary decoder model is good for high-FMS examples. More experiments on Chinese-English, English-German and English-Spanish translation demonstrate that the combination of the two EGNMT models can achieve improvements of up to +9 BLEU points over the baseline system and +7 BLEU points over a two-encoder Transformer.

Motivation & Objective

• To enable neural machine translation (NMT) to effectively reuse translation segments from similar example sentences.
• To address the challenge of distinguishing reusable translation fragments from noisy, mismatched content in example translations.
• To develop a trainable, end-to-end framework that integrates example guidance into the NMT decoding process without altering the core architecture.
• To improve translation quality by leveraging both low- and high-similarity example translations through complementary modeling strategies.

Proposed method

• A noise-masked encoder (NME) model uses word alignments to identify and mask out non-relevant words in example translations with a special token <X>, then encodes the cleaned sequence.
• An auxiliary decoder (AD) model shares parameters with the primary decoder and is trained to predict a masked version of the reference translation, where non-matching parts are replaced with <X>.
• The auxiliary decoder is jointly trained with the primary decoder, enabling the primary model to learn to generate reusable phrases from the example translation.
• The framework is implemented over the Transformer architecture and supports any encoder-decoder-based NMT system.
• The NME and AD models are combined to leverage their complementary strengths: NME excels with low-fuzzy-match-score (FMS) examples by filtering noise, while AD performs better on high-FMS examples by directly teaching reuse.

Experimental results

Research questions

• RQ1 Can NMT be effectively guided to reuse relevant translation fragments from example sentences while filtering out noisy, mismatched content?
• RQ2 How can the primary NMT decoder be trained to learn from example translations without modifying its architecture?
• RQ3 What is the relative contribution of noise masking versus auxiliary decoding in improving translation quality across varying example similarity levels?
• RQ4 Can a unified framework combine both noise masking and auxiliary decoding to outperform baselines and single-model approaches?

Key findings

• The EGNMT framework achieves up to +9.0 BLEU points improvement over the baseline Transformer model on Chinese-English, English-German, and English-Spanish translation tasks.
• The auxiliary decoder model (AD) achieves the highest performance on high-fuzzy-match-score (FMS) examples, while the noise-masked encoder (NME) performs best on low-FMS examples.
• The combined EGNMT model achieves the highest F1 score for reusable word generation across all FMS levels, demonstrating effective balance between precision and recall.
• Attention visualization confirms that the final model correctly focuses on reusable translation fragments in the example, unlike the baseline which attends randomly across the example.
• The model filters out 8.1% of words in example translations due to incorrect word alignments, but still maintains high reuse accuracy.
• The framework is general and can be applied to any Transformer-based NMT system, with potential for integration with translation memory or domain adaptation.