[Paper Review] Context Gates for Neural Machine Translation

Zhaopeng Tu, Yang Liu+3 more|arXiv (Cornell University)|Aug 22, 2016

Natural Language Processing Techniques19 references15 citations

TL;DR

This paper proposes context gates in neural machine translation (NMT) to dynamically balance the influence of source and target contexts during decoding, improving translation adequacy without sacrificing fluency. By learning to weight source and target context contributions per decoding step, the method achieves a +2.3 BLEU point gain over standard attention-based NMT, with stronger gains on long sentences and improved robustness in handling content vs. function words.

ABSTRACT

In neural machine translation (NMT), generation of a target word depends on both source and target contexts. We find that source contexts have a direct impact on the adequacy of a translation while target contexts affect the fluency. Intuitively, generation of a content word should rely more on the source context and generation of a functional word should rely more on the target context. Due to the lack of effective control over the influence from source and target contexts, conventional NMT tends to yield fluent but inadequate translations. To address this problem, we propose context gates which dynamically control the ratios at which source and target contexts contribute to the generation of target words. In this way, we can enhance both the adequacy and fluency of NMT with more careful control of the information flow from contexts. Experiments show that our approach significantly improves upon a standard attention-based NMT system by +2.3 BLEU points.

Motivation & Objective

• To address the issue of NMT producing fluent but inadequate translations due to over-reliance on target context.
• To investigate whether source and target contexts have distinct roles in translation adequacy and fluency.
• To design a mechanism that dynamically controls context contributions based on word type (content vs. function words).
• To improve translation quality by enhancing the system's ability to balance adequacy and fluency through learnable context gating.
• To demonstrate that context gates can replace GRUs with simpler RNNs while maintaining performance and improving decoding speed.

Proposed method

• Introduce context gates as differentiable, non-linear gating units that output a ratio between 0 and 1 to control the contribution of source and target contexts at each decoding step.
• Compute the context gate output $ z_t $ using a gated mechanism that takes both source context $ oldsymbol{h}_s $ and target context $ oldsymbol{h}_t $ as inputs.
• Use the gate output $ z_t $ to interpolate between source and target context representations: $ oldsymbol{h}_t^{ ext{gate}} = z_t oldsymbol{h}_s + (1 - z_t) oldsymbol{h}_t $.
• Integrate the gated context representation into the decoder RNN (GRU or RNN) to generate the next target word.
• Train the model end-to-end with standard NMT objectives, where the gate parameters are learned from data.
• Replace GRU units with standard RNNs in the decoder while using context gates, reducing parameters by half and halving matrix computation costs.

Experimental results

Research questions

• RQ1 Does the relative influence of source and target contexts correlate with translation adequacy and fluency, respectively?
• RQ2 Can dynamically controlling the balance between source and target context improve translation quality in NMT?
• RQ3 Does the model benefit more from increased source context influence on long or complex sentences?
• RQ4 Can context gates replace GRUs while maintaining performance and improving inference speed?
• RQ5 Is the context gate's output correlated with translation quality, especially for adequacy and fluency?

Key findings

• Context gates improve translation performance by +2.3 BLEU points over a standard attention-based NMT system.
• The method significantly enhances translation adequacy, especially on long sentences (≥30 tokens), where performance degradation in baseline models is alleviated.
• The context gate weight $ z_t $ is positively correlated with translation improvement, with a correlation of 0.076 for sentences of length ≥30.
• Context gates allow the replacement of GRU units with simpler RNNs, reducing model parameters by half and cutting matrix computation costs by 50% during decoding.
• The system maintains high fluency while improving adequacy, as shown by reduced over-translation and under-translation errors in qualitative examples.
• The correlation between gate weight and translation quality is higher on long sentences, indicating that source context is more critical for complex or lengthy source inputs.