[论文解读] Why Neural Machine Translation Prefers Empty Outputs

Xing Shi, Yijun Xiao+1 more|arXiv (Cornell University)|Dec 24, 2020

Natural Language Processing Techniques参考文献 21被引用 5

一句话总结

本文 identifies two root causes for neural machine translation (NMT) systems favoring empty outputs: label smoothing reduces confidence in correct-length translations, and the use of a single, shared End-of-Sequence (EoS) token across all sentence lengths creates an implicit smoothing effect that boosts the probability of zero-length translations. By introducing distinct EoS tokens for different target lengths (MultiEoS), the authors eliminate this bias, drastically reducing empty translation rates while restoring proper length regulation.

ABSTRACT

We investigate why neural machine translation (NMT) systems assign high probability to empty translations. We find two explanations. First, label smoothing makes correct-length translations less confident, making it easier for the empty translation to finally outscore them. Second, NMT systems use the same, high-frequency EoS word to end all target sentences, regardless of length. This creates an implicit smoothing that increases zero-length translations. Using different EoS types in target sentences of different lengths exposes and eliminates this implicit smoothing.

研究动机与目标

• To investigate why NMT systems assign high probability to empty translations despite no such examples in training data.
• To analyze the role of label smoothing in reducing confidence for correct-length translations.
• To examine how the use of a single EoS token across all sentence lengths creates an implicit smoothing effect favoring zero-length outputs.
• To evaluate whether using distinct EoS tokens for different target lengths can eliminate this bias and improve translation length accuracy.

提出的方法

• The authors analyze the probability distribution P(y|x) in NMT models, focusing on the role of the EoS token in decoding.
• They introduce a MultiEoS setup where each target sentence of length l ends with a unique [EOS-l] token, replacing the universal [EOS] token.
• They train models with and without label smoothing, comparing the log-probability of [EoS] at the first decoding step across different language pairs.
• They compute the perplexity of the target length distribution Q(l|m) to quantify uncertainty in length prediction.
• They use beam search with beam size 512 to decode test sets and measure empty translation ratios and length ratios.
• They compare SingleEoS (standard) and MultiEoS setups to isolate the effect of implicit smoothing from label smoothing.

实验结果

研究问题

• RQ1 Why do NMT systems assign high probability to empty translations despite no such examples in the training data?
• RQ2 How does label smoothing contribute to the preference for short or empty translations?
• RQ3 What role does the use of a single EoS token across all sentence lengths play in increasing the relative probability of zero-length outputs?
• RQ4 Can using different EoS tokens for different target lengths eliminate the implicit smoothing effect and reduce empty translation rates?

主要发现

• Label smoothing reduces the confidence of correct-length translations by approximately 5% per token, cumulatively making empty translations more probable.
• The single-EoS smoothing effect increases the probability of zero-length translations by implicitly distributing probability mass toward unseen lengths, especially when the target length distribution Q(l|m) has high uncertainty.
• In the MultiEoS setup, the log-probability of [EoS] at the first position drops to near the minimum among all words, indicating it is no longer favored as a generation candidate.
• With MultiEoS, the empty translation ratio drops from 52.5% (EN→DE) and 52.0% (DE→EN) to 2.5% and 2.0%, respectively, while length ratios improve from 0.26 and 0.31 to 0.74 and 0.86.
• The perplexity of Q(l|m) correlates with empty translation rates: higher perplexity (greater uncertainty) leads to higher empty ratios, which is mitigated by MultiEoS.
• The MultiEoS approach eliminates the preference for empty outputs without requiring additional length regularization, demonstrating that the issue stems from architectural design rather than training objective.