[论文解读] Mutual Information and Diverse Decoding Improve Neural Machine Translation

Jiwei Li, Dan Jurafsky|arXiv (Cornell University)|Jan 4, 2016

Natural Language Processing Techniques参考文献 39被引用 98

一句话总结

本论文通过重排序引入信息互信息目标以及促进多样性的解码方法，以改进神经机器翻译，在 WMT14 EN-DE 和 EN-FR 上对标准和基于注意力的模型显示 BLEU 增益。

ABSTRACT

Sequence-to-sequence neural translation models learn semantic and syntactic relations between sentence pairs by optimizing the likelihood of the target given the source, i.e., $p(y|x)$, an objective that ignores other potentially useful sources of information. We introduce an alternative objective function for neural MT that maximizes the mutual information between the source and target sentences, modeling the bi-directional dependency of sources and targets. We implement the model with a simple re-ranking method, and also introduce a decoding algorithm that increases diversity in the N-best list produced by the first pass. Applied to the WMT German/English and French/English tasks, the proposed models offers a consistent performance boost on both standard LSTM and attention-based neural MT architectures.

研究动机与目标

• Motivate and address limitations of unidirectional p(y|x) training in Seq2Seq MT.
• Model and evaluate a mutual information objective that incorporates p(x|y) to capture bidirectional dependencies.
• Improve decoding diversity to yield more varied high-quality translations.
• Demonstrate empirical gains on WMT14 EN→DE, EN→FR, and DE→EN with both standard and attention-based NMT architectures.

提出的方法

• Train two separate Seq2Seq models to learn p(y|x) and p(x|y).
• Generate N-best lists from p(y|x) using beam search and rerank them with log p(x|y).
• Promote diversity in the first-pass N-best list by a diversity-aware beam search (penalizing lower-ranked siblings).
• Rerank final candidates by a linear combination of log p(y|x), log p(x|y), log p(y), and target length to optimize BLEU via MERT.
• Apply unknown word replacement using alignments from attention models to improve UNK handling.
• Evaluate on WMT’14 EN↔DE and EN↔FR with both standard Seq2Seq and attention-based models.

实验结果

研究问题

• RQ1 Does incorporating p(x|y) via a mutual information objective improve translation quality over standard p(y|x) models?
• RQ2 Can a diversity-promoting decoding strategy yield more diverse and high-quality translation hypotheses for reranking?
• RQ3 What is the impact of mutual information reranking and diversity decoding across different architectures (standard vs. attention-based) and language pairs (EN-DE, EN-FR, DE-EN)?

主要发现

• Reranking with mutual information (p(y|x) and p(x|y)) improves BLEU over standard models across tasks.
• Diversity-promoting decoding increases N-best list diversity (distinct-1 and distinct-2) and contributes additional BLEU gains.
• Unknown word replacement significantly boosts BLEU, especially in EN-DE and EN-FR results.
• Diversity decoding plus MI reranking yields overall gains of up to approximately +2.1 to +2.6 BLEU points over standard models for EN-DE and EN-FR.
• Attention-based models with MI and diversity decoding reach the highest BLEU scores reported in the study (e.g., up to 36.3 in FR→EN with all features).