[論文レビュー] Mutual Information and Diverse Decoding Improve Neural Machine Translation

Jiwei Li, Dan Jurafsky|arXiv (Cornell University)|Jan 4, 2016

Natural Language Processing Techniques参考文献 39被引用数 98

ひとこと要約

この論文はリランキングを介した相互情報量目的と多様性を促進するデコーディング法を導入し、ニューラル機械翻訳を改善し、標準モデルとアテンションベースモデルの両方でWMT14 EN-DEおよび EN-FRにおいて BLEU の向上を示す。

ABSTRACT

Sequence-to-sequence neural translation models learn semantic and syntactic relations between sentence pairs by optimizing the likelihood of the target given the source, i.e., $p(y|x)$, an objective that ignores other potentially useful sources of information. We introduce an alternative objective function for neural MT that maximizes the mutual information between the source and target sentences, modeling the bi-directional dependency of sources and targets. We implement the model with a simple re-ranking method, and also introduce a decoding algorithm that increases diversity in the N-best list produced by the first pass. Applied to the WMT German/English and French/English tasks, the proposed models offers a consistent performance boost on both standard LSTM and attention-based neural MT architectures.

研究の動機と目的

• Motivate and address limitations of unidirectional p(y|x) training in Seq2Seq MT.
• Model and evaluate a mutual information objective that incorporates p(x|y) to capture bidirectional dependencies.
• Improve decoding diversity to yield more varied high-quality translations.
• Demonstrate empirical gains on WMT14 EN→DE, EN→FR, and DE→EN with both standard and attention-based NMT architectures.

提案手法

• Train two separate Seq2Seq models to learn p(y|x) and p(x|y).
• Generate N-best lists from p(y|x) using beam search and rerank them with log p(x|y).
• Promote diversity in the first-pass N-best list by a diversity-aware beam search (penalizing lower-ranked siblings).
• Rerank final candidates by a linear combination of log p(y|x), log p(x|y), log p(y), and target length to optimize BLEU via MERT.
• Apply unknown word replacement using alignments from attention models to improve UNK handling.
• Evaluate on WMT’14 EN↔DE and EN↔FR with both standard Seq2Seq and attention-based models.

実験結果

リサーチクエスチョン

• RQ1 Does incorporating p(x|y) via a mutual information objective improve translation quality over standard p(y|x) models?
• RQ2 Can a diversity-promoting decoding strategy yield more diverse and high-quality translation hypotheses for reranking?
• RQ3 What is the impact of mutual information reranking and diversity decoding across different architectures (standard vs. attention-based) and language pairs (EN-DE, EN-FR, DE-EN)?

主な発見

Model	Features	BLEU scores
Standard	p(y|x)	13.2
Standard	p(y|x) + Length	13.6 (+0.4)
Standard	p(y|x) + p(x|y) + Length	15.0 (+1.4)
Standard	p(y|x) + p(x|y) + p(y) + Length	15.4 (+0.4)
Standard	p(y|x) + p(x|y) + p(y) + Length + Diver decoding	15.8 (+0.4)
Standard+ UnkRep	p(y|x)	14.7
Standard+ UnkRep	p(y|x) + Length	15.2 (+0.7)
Standard+ UnkRep	p(y|x) + p(x|y) + Length	16.3 (+1.1)
Standard+ UnkRep	p(y|x) + p(x|y) + p(y) + Length	16.7 (+0.4)
Standard+ UnkRep	p(y|x) + p(x|y) + p(y) + Length + Diver decoding	17.3 (+0.3)
Attention+ UnkRep	p(y|x)	20.5
Attention+ UnkRep	p(y|x) + Length	20.9 (+0.4)
Attention+ UnkRep	p(y|x) + p(x|y) + Length	21.8 (+0.9)
Attention+ UnkRep	p(y|x) + p(x|y) + p(y) + Length	22.1 (+0.3)
Attention+ UnkRep	p(y|x) + p(x|y) + p(y) + Length + Diver decoding	22.6 (+0.3)

• Reranking with mutual information (p(y|x) and p(x|y)) improves BLEU over standard models across tasks.
• Diversity-promoting decoding increases N-best list diversity (distinct-1 and distinct-2) and contributes additional BLEU gains.
• Unknown word replacement significantly boosts BLEU, especially in EN-DE and EN-FR results.
• Diversity decoding plus MI reranking yields overall gains of up to approximately +2.1 to +2.6 BLEU points over standard models for EN-DE and EN-FR.
• Attention-based models with MI and diversity decoding reach the highest BLEU scores reported in the study (e.g., up to 36.3 in FR→EN with all features).