[Paper Review] Analyzing Uncertainty in Neural Machine Translation

Myle Ott, Michael Auli+2 more|arXiv (Cornell University)|Feb 28, 2018

Natural Language Processing Techniques30 references55 citations

TL;DR

The paper analyzes intrinsic and extrinsic uncertainty in neural machine translation, compares beam search and sampling, and shows that while search is effective, models spread probability mass too broadly and training data noise (like copied sources) degrades large-beam performance; it also offers calibration analyses and simple mitigation strategies.

ABSTRACT

Machine translation is a popular test bed for research in neural sequence-to-sequence models but despite much recent research, there is still a lack of understanding of these models. Practitioners report performance degradation with large beams, the under-estimation of rare words and a lack of diversity in the final translations. Our study relates some of these issues to the inherent uncertainty of the task, due to the existence of multiple valid translations for a single source sentence, and to the extrinsic uncertainty caused by noisy training data. We propose tools and metrics to assess how uncertainty in the data is captured by the model distribution and how it affects search strategies that generate translations. Our results show that search works remarkably well but that models tend to spread too much probability mass over the hypothesis space. Next, we propose tools to assess model calibration and show how to easily fix some shortcomings of current models. As part of this study, we release multiple human reference translations for two popular benchmarks.

Motivation & Objective

• Investigate how intrinsic task uncertainty (multiple valid translations) and extrinsic data noise affect NMT performance.
• Assess how well model distributions fit the data distribution and how this impacts search strategies.
• Characterize calibration at token, set, and sequence levels and identify remedies.
• Evaluate the impact of training data artifacts (e.g., copied sources) on large-beam decoding.
• Provide practical mitigation strategies and release human reference translations for benchmarks.

Proposed method

• Use pre-trained Fairseq-based sequence-to-sequence models with encoder–decoder architecture and attention.
• Quantify uncertainty by comparing beam search and sampling across hypotheses and evaluating BLEU and model likelihood.
• Analyze token-level unigram statistics to compare model outputs with data distributions.
• Assess set-level calibration by comparing model probabilities over sets of hypotheses to true data probabilities.
• Introduce synthetic copy-noise into training data to study extrinsic uncertainty effects and test mitigation strategies.
• Release multiple human reference translations for WMT datasets to support evaluation.

Experimental results

Research questions

• RQ1 How well do NMT models capture the inherent ambiguity of translation versus artifacts of training data?
• RQ2 Why do large beam widths degrade translation quality, and is this due to model fit or search bias?
• RQ3 Do NMT models calibrate probability estimates at token, sentence, and set levels?
• RQ4 What is the impact of data artifacts such as copied-source targets on decoding behavior and translation quality?
• RQ5 Can simple data cleaning and inference constraints mitigate issues observed with wide beams?

Key findings

• Beam search is effective at finding high-likelihood translations, but the model distribution spreads probability mass too widely across hypotheses.
• Sampling yields diverse outputs with lower top-BLEU than beam search for the top candidate, and BLEU can decrease as more samples are drawn.
• Copy-like noise in training data (target copies of source) disproportionately inflates copies in large-beam outputs, degrading BLEU for wide beams.
• Data preprocessing (removing copy-like pairs) and inference constraints (pruning overlaps with source) mitigate large-beam degradation and improve BLEU.
• Unigram statistics from model samples align with data for common words but under-represent rare words in beam outputs, indicating calibration gaps at the word level.
• Set-level calibration shows model probability mass aligns with data distribution when considering sets of hypotheses, even as individual sequences may be over- or under-estimated.