[Paper Review] Merging External Bilingual Pairs into Neural Machine Translation

Tao Wang, Shaohui Kuang+2 more|arXiv (Cornell University)|Dec 2, 2019

Natural Language Processing Techniques22 references6 citations

TL;DR

This paper proposes a data-centric approach to integrate pre-specified bilingual phrase pairs into neural machine translation (NMT) by modifying training data through tagging, mixed phrases, and extra embeddings—achieving over 99% accuracy in translating specified phrases without altering the NMT architecture or decoding speed.

ABSTRACT

As neural machine translation (NMT) is not easily amenable to explicit correction of errors, incorporating pre-specified translations into NMT is widely regarded as a non-trivial challenge. In this paper, we propose and explore three methods to endow NMT with pre-specified bilingual pairs. Instead, for instance, of modifying the beam search algorithm during decoding or making complex modifications to the attention mechanism --- mainstream approaches to tackling this challenge ---, we experiment with the training data being appropriately pre-processed to add information about pre-specified translations. Extra embeddings are also used to distinguish pre-specified tokens from the other tokens. Extensive experimentation and analysis indicate that over 99% of the pre-specified phrases are successfully translated (given a 85% baseline) and that there is also a substantive improvement in translation quality with the methods explored here.

Motivation & Objective

• To address the challenge of integrating pre-specified bilingual phrase pairs into end-to-end NMT systems, which are not easily amenable to explicit error correction.
• To enable NMT to reliably translate specific multi-word expressions (e.g., brand names) as defined in an external database, ensuring high accuracy and consistency.
• To develop a method that improves translation quality for pre-specified phrases without modifying the NMT model architecture or beam search algorithm.
• To maintain decoding speed while significantly increasing the success rate of translating fixed bilingual expressions.
• To explore a scalable, reproducible, and effective way to inject external linguistic knowledge into NMT via training data preprocessing.

Proposed method

• Tagging method: Enclose source-side phrases and their target translations with special tokens <start> and <end> to create a pattern that the NMT model can learn to associate.
• Mixed phrase method: Augment the source sentence by appending the target translation of a phrase directly after its source form (e.g., '我 爱 香港 hong kong'), enhancing copy learning.
• Extra embedding method: Introduce a dedicated embedding vector to distinguish pre-specified phrases from regular tokens, providing a softer signal than tagging.
• Combined approach: Use tagging and mixed phrases together, with optional <middle> markers to separate source and target in mixed phrases, improving alignment learning.
• Shared embeddings across tags and target phrases help the model learn cross-lingual correspondence through stronger attention signals.
• All methods are applied during data preprocessing—no changes to NMT model or decoding algorithm are required, preserving inference speed.

Experimental results

Research questions

• RQ1 Can NMT be effectively trained to translate specific pre-specified bilingual phrases with high accuracy using only data-level modifications?
• RQ2 Does combining tagging, mixed phrases, and extra embeddings lead to a significant improvement in phrase translation success rate compared to baseline NMT?
• RQ3 Can the proposed method achieve high accuracy for both word-level and multi-word phrase translations without degrading decoding speed?
• RQ4 How do the individual components (tagging, mixed phrases, extra embeddings) contribute to the overall performance gain in phrase translation?
• RQ5 Is the method robust across different language pairs and domains, such as Chinese-English and English-German news translation?

Key findings

• On Chinese-to-English translation, the success rate for pre-specified phrases increased from 74.19% (baseline) to 98.40% at the sentence level and from 85.03% to 99.13% at the phrase level.
• The method achieved a 1.4 BLEU point improvement over the baseline (45.7 BLEU) on Chinese-to-English translation.
• On English-to-German translation, the phrase-level success rate improved from 93.69% (baseline) to 99.54% using the proposed methods.
• The tagging method enhanced attention mechanisms by creating a learnable pattern for copying, while shared embeddings strengthened cross-lingual alignment.
• The extra embedding method provided a softer, less disruptive signal than tagging, though with a smaller performance gain.
• The approach achieved over 99% success rate in translating pre-specified phrases without modifying the NMT model or decoding process, preserving inference speed.