[Paper Review] Modeling Coverage for Neural Machine Translation

Zhaopeng Tu, Zhengdong Lu+3 more|arXiv (Cornell University)|Jan 19, 2016

Natural Language Processing Techniques27 references159 citations

TL;DR

Introduces a coverage mechanism for NMT to track attention history; uses linguistic and NN-based coverage to reduce under-/over-translation and improve alignment, with BLEU gains on Chinese–English.

ABSTRACT

Attention mechanism has enhanced state-of-the-art Neural Machine Translation (NMT) by jointly learning to align and translate. It tends to ignore past alignment information, however, which often leads to over-translation and under-translation. To address this problem, we propose coverage-based NMT in this paper. We maintain a coverage vector to keep track of the attention history. The coverage vector is fed to the attention model to help adjust future attention, which lets NMT system to consider more about untranslated source words. Experiments show that the proposed approach significantly improves both translation quality and alignment quality over standard attention-based NMT.

Motivation & Objective

• Motivate and address the lack of explicit coverage in attention-based NMT leading to over- and under-translation.
• Propose a coverage mechanism that maintains a coverage vector updated after each attention step.
• Explore linguistic and neural network-based coverage models to integrate into the NMT attention mechanism.

Proposed method

• Maintain a per-source-word coverage vector C_{i-1} that summarizes past attention for each source word.
• Integrate the coverage vector into the attention model to adjust future attention via e_{i,j} = a(t_{i-1}, h_j, C_{i-1,j}).
• Propose linguistic coverage with a scalar or fertility-based normalization Phi_j to compute coverage C_{i,j} as C_{i,j}=C_{i-1,j}+ (1/Phi_j) * alpha_{i,j} and explore pre-computed fertility Phi_j.
• Propose neural-network-based coverage where C_{i,j} is updated through a function f(C_{i-1,j}, alpha_{i,j}, h_j, t_{i-1}) with GRU-style gating.
• Train end-to-end to maximize P(y|x; theta, eta) and compare with baseline attention-based NMT (GroundHog) and Moses.
• Evaluate both translation quality via BLEU and alignment quality via SAER/AER, with a focus on long sentences.

Experimental results

Research questions

• RQ1 Does incorporating a coverage mechanism into attention-based NMT improve translation quality compared to standard attention-based NMT and SMT baselines?
• RQ2 How do linguistic coverage and neural network-based coverage variants perform in terms of translation quality and alignment accuracy?
• RQ3 What is the impact of coverage on long sentences and under-/over-translation phenomena?

Key findings

• Coverage-based NMT significantly improves BLEU scores over standard attention-based NMT on MT05, MT06, and MT08 datasets.
• Linguistic coverage with fertility yields notable BLEU gains and improves alignment, with fertility helping estimate covered ratios.
• NN-based coverage with gating also improves BLEU, with higher dimensional coverage (d=10) providing additional gains.
• Coverage reduces under-translation and over-translation in subjective evaluations, and improves translation adequacy and fluency.
• Coverage helps mitigate performance drop on longer sentences, pushing attention toward untranslated words and lengthening translations appropriately.
• The most complex NN-based coverage adds relatively few parameters yet delivers competitive improvements.