[Paper Review] Mutual Information and Diverse Decoding Improve Neural Machine Translation

Jiwei Li, Dan Jurafsky|arXiv (Cornell University)|Jan 4, 2016

Natural Language Processing Techniques39 references98 citations

TL;DR

This paper introduces a mutual information objective via reranking and a diversity-promoting decoding method to improve neural machine translation, showing BLEU gains on WMT14 EN-DE and EN-FR across standard and attention-based models.

ABSTRACT

Sequence-to-sequence neural translation models learn semantic and syntactic relations between sentence pairs by optimizing the likelihood of the target given the source, i.e., $p(y|x)$, an objective that ignores other potentially useful sources of information. We introduce an alternative objective function for neural MT that maximizes the mutual information between the source and target sentences, modeling the bi-directional dependency of sources and targets. We implement the model with a simple re-ranking method, and also introduce a decoding algorithm that increases diversity in the N-best list produced by the first pass. Applied to the WMT German/English and French/English tasks, the proposed models offers a consistent performance boost on both standard LSTM and attention-based neural MT architectures.

Motivation & Objective

• Motivate and address limitations of unidirectional p(y|x) training in Seq2Seq MT.
• Model and evaluate a mutual information objective that incorporates p(x|y) to capture bidirectional dependencies.
• Improve decoding diversity to yield more varied high-quality translations.
• Demonstrate empirical gains on WMT14 EN→DE, EN→FR, and DE→EN with both standard and attention-based NMT architectures.

Proposed method

• Train two separate Seq2Seq models to learn p(y|x) and p(x|y).
• Generate N-best lists from p(y|x) using beam search and rerank them with log p(x|y).
• Promote diversity in the first-pass N-best list by a diversity-aware beam search (penalizing lower-ranked siblings).
• Rerank final candidates by a linear combination of log p(y|x), log p(x|y), log p(y), and target length to optimize BLEU via MERT.
• Apply unknown word replacement using alignments from attention models to improve UNK handling.
• Evaluate on WMT’14 EN↔DE and EN↔FR with both standard Seq2Seq and attention-based models.

Experimental results

Research questions

• RQ1 Does incorporating p(x|y) via a mutual information objective improve translation quality over standard p(y|x) models?
• RQ2 Can a diversity-promoting decoding strategy yield more diverse and high-quality translation hypotheses for reranking?
• RQ3 What is the impact of mutual information reranking and diversity decoding across different architectures (standard vs. attention-based) and language pairs (EN-DE, EN-FR, DE-EN)?

Key findings

• Reranking with mutual information (p(y|x) and p(x|y)) improves BLEU over standard models across tasks.
• Diversity-promoting decoding increases N-best list diversity (distinct-1 and distinct-2) and contributes additional BLEU gains.
• Unknown word replacement significantly boosts BLEU, especially in EN-DE and EN-FR results.
• Diversity decoding plus MI reranking yields overall gains of up to approximately +2.1 to +2.6 BLEU points over standard models for EN-DE and EN-FR.
• Attention-based models with MI and diversity decoding reach the highest BLEU scores reported in the study (e.g., up to 36.3 in FR→EN with all features).