[Paper Review] ngram-OAXE: Phrase-Based Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation

Cunxiao Du, Zhaopeng Tu+2 more|arXiv (Cornell University)|Oct 8, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes ngram-OaXE, a phrase-based order-agnostic cross entropy loss for non-autoregressive machine translation that improves translation quality by allowing reordering only between n-gram phrases while preserving strict word order within each phrase. By modeling ngram-level probability distributions and optimizing over the best ngram ordering via the Hungarian algorithm, ngram-OaXE reduces repetition and improves fluency, achieving up to +3.8 BLEU points over OaXE on raw WMT14 En-De data without knowledge distillation.

ABSTRACT

Recently, a new training oaxe loss has proven effective to ameliorate the effect of multimodality for non-autoregressive translation (NAT), which removes the penalty of word order errors in the standard cross-entropy loss. Starting from the intuition that reordering generally occurs between phrases, we extend oaxe by only allowing reordering between ngram phrases and still requiring a strict match of word order within the phrases. Extensive experiments on NAT benchmarks across language pairs and data scales demonstrate the effectiveness and universality of our approach. %Further analyses show that the proposed ngram-oaxe alleviates the multimodality problem with a better modeling of phrase translation. Further analyses show that ngram-oaxe indeed improves the translation of ngram phrases, and produces more fluent translation with a better modeling of sentence structure.

Motivation & Objective

• To address the multimodality problem in non-autoregressive translation (NAT), where independent token predictions lead to repetitive and non-fluent outputs.
• To improve NAT performance on raw, undistilled training data by modeling reordering at the ngram phrase level rather than at the individual word level.
• To maintain fast decoding speed while enhancing phrase-level translation accuracy and sentence structure modeling.
• To reduce reliance on knowledge distillation by enabling effective training from raw data through a more structured training objective.

Proposed method

• The method constructs ngram probability distributions by multiplying individual word probabilities from the NAT model over contiguous n-gram spans.
• It formulates the ngram-OaXE loss as the minimum cross-entropy over all possible permutations of ngram phrases in the target sentence.
• The optimal ngram ordering is computed efficiently using the Hungarian algorithm to minimize the loss.
• The approach is implemented with minimal code changes—adding only one line to the OaXE source code—ensuring marginal training time increase (e.g., 3%).
• The method preserves strict word order within each ngram while allowing reordering between ngrams, aligning better with syntactic and grammatical constraints.
• The loss function is applied during training, enabling the model to learn more robust and fluent translations without requiring autoregressive decoding.

Experimental results

Research questions

• RQ1 Can modeling reordering at the ngram level instead of the word level improve non-autoregressive translation quality?
• RQ2 Does ngram-OaXE reduce token repetition and improve fluency compared to OaXE and standard cross-entropy loss?
• RQ3 Can ngram-OaXE enable strong NAT performance on raw, undistilled training data, reducing the need for knowledge distillation?
• RQ4 To what extent does ngram-OaXE improve the modeling of long sentences and phrase-level coherence?

Key findings

• ngram-OaXE improves translation performance over OaXE by up to +3.8 BLEU points on raw WMT14 En-De data, demonstrating strong gains without knowledge distillation.
• On distilled data, ngram-OaXE reduces token repetition from 1.56% (OaXE) to 0.98%, indicating improved handling of multimodality.
• The model achieves lower perplexity scores, indicating better fluency in generated translations compared to OaXE across all settings.
• ngram-OaXE significantly improves the accuracy of ngram phrase generation, especially for longer and more complex phrases.
• The method narrows the performance gap between training on raw data and distilled data, suggesting it enables effective self-supervised learning in NAT.
• The training time increase is minimal—only about 3% more than OaXE—making it practical for large-scale training.