[Paper Review] Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
Proposes a differentiable Bag-of-N-grams (BoN) objective to train non-autoregressive NMT (NAT), aiming to better model target-side sequential dependency and improve translation quality, especially for longer sentences.
Non-Autoregressive Neural Machine Translation (NAT) achieves significant decoding speedup through generating target words independently and simultaneously. However, in the context of non-autoregressive translation, the word-level cross-entropy loss cannot model the target-side sequential dependency properly, leading to its weak correlation with the translation quality. As a result, NAT tends to generate influent translations with over-translation and under-translation errors. In this paper, we propose to train NAT to minimize the Bag-of-Ngrams (BoN) difference between the model output and the reference sentence. The bag-of-ngrams training objective is differentiable and can be efficiently calculated, which encourages NAT to capture the target-side sequential dependency and correlates well with the translation quality. We validate our approach on three translation tasks and show that our approach largely outperforms the NAT baseline by about 5.0 BLEU scores on WMT14 En$\leftrightarrow$De and about 2.5 BLEU scores on WMT16 En$\leftrightarrow$Ro.
Motivation & Objective
- Motivate NAT by addressing the poor correlation between word-level cross-entropy and translation quality.
- Introduce a differentiable BoN-based objective to model target-side sequential dependencies in NAT.
- Provide an efficient computation method for BoN within NAT so training remains fast.
- Demonstrate that BoN-based training improves translation quality across multiple language pairs.
- Show that BoN-based methods can be used standalone (fine-tuning) or jointly with cross-entropy for end-to-end training.
Proposed method
- Define BoN for a discrete sentence as a sum of n-gram one-hot vectors.
- Define BoN for NAT as the expectation of BoN over all possible translations, enabling an efficient, per-position decomposition.
- Derive an efficient computation of BoN by sliding a window over NAT output distributions and assuming target tokens at each position are independent.
- Propose BoN loss as BoN- L1 distance between NAT’s BoN and the reference BoN, leveraging sparsity of reference BoN and the fact that BoN-θ sums over many possible translations.
- Formulate training objectives: BoN-FT (BoN fine-tuning), BoN-Joint (α-weighted combination of CE and BoN losses), and BoN-Joint+FT (BoN-Joint followed by BoN fine-tuning).
- Experiment with n-gram sizes (n=1..4) and α values to study correlations with BLEU and overall performance.
Experimental results
Research questions
- RQ1Can BoN-based sequence-level objectives better correlate with translation quality than cross-entropy for NAT?
- RQ2Does BoN training reduce common NAT errors like over- and under-translation and repeated tokens, especially on longer sentences?
- RQ3How do joint BoN and CE objectives compare to BoN-only fine-tuning for NAT in terms of BLEU gains and training speed?
- RQ4What is the impact of different n-gram sizes on BoN effectiveness for NAT?
Key findings
- BoN-based objectives yield higher correlation with translation quality than cross-entropy, especially on longer sentences.
- BoN-FT achieves faster training and improves over NAT-Base, often outperforming Reinforce-NAT in BLEU while remaining much faster to train.
- BoN-Joint substantially improves BLEU over BoN-FT, and BoN-Joint+FT achieves the largest gains (e.g., around 5.0 BLEU on WMT14 En↔De and 2.5 BLEU on WMT16 En↔Ro in their setups).
- BoN with n>1 (notably n=2) provides better correlation and BLEU gains than n=1, indicating the importance of modeling short sequential dependencies.
- BoN methods reduce repeated tokens and overcorrection, particularly on longer sentences, as shown by postprocessing analyses.
- BoN-Joint converges faster than some baselines and maintains competitive training speed due to the differentiable BoN objective.
Better researchstarts right now
From reading papers to final review, dramatically reduce your research time.
No credit card · Free plan available
This review was created by AI and reviewed by human editors.