[Paper Review] Understanding and Improving Lexical Choice in Non-Autoregressive Translation

Liang Ding, Longyue Wang+4 more|arXiv (Cornell University)|Dec 29, 2020

Natural Language Processing TechniquesComputer Science40 references44 citations

TL;DR

The paper analyzes how knowledge distillation from autoregressive teachers biases NAT lexical choices, especially for low-frequency words, and introduces raw-data–informed priors via KL divergence to improve lexical accuracy and BLEU across language pairs.

ABSTRACT

Knowledge distillation (KD) is essential for training non-autoregressive translation (NAT) models by reducing the complexity of the raw data with an autoregressive teacher model. In this study, we empirically show that as a side effect of this training, the lexical choice errors on low-frequency words are propagated to the NAT model from the teacher model. To alleviate this problem, we propose to expose the raw data to NAT models to restore the useful information of low-frequency words, which are missed in the distilled data. To this end, we introduce an extra Kullback-Leibler divergence term derived by comparing the lexical choice of NAT model and that embedded in the raw data. Experimental results across language pairs and model architectures demonstrate the effectiveness and universality of the proposed approach. Extensive analyses confirm our claim that our approach improves performance by reducing the lexical choice errors on low-frequency words. Encouragingly, our approach pushes the SOTA NAT performance on the WMT14 English-German and WMT16 Romanian-English datasets up to 27.8 and 33.8 BLEU points, respectively. The source code will be released.

Motivation & Objective

• Identify how knowledge distillation affects lexical choice in NAT models, with a focus on low-frequency words.
• Quantify lexical choice using a new AoLC metric across language pairs.
• Propose data-dependent priors derived from raw data to preserve useful low-frequency lexical information.
• Develop and evaluate a KL-based training objective integrating raw-data priors with standard NAT training.
• Demonstrate robustness across NAT architectures and multiple datasets.

Proposed method

• Define AoLC (accuracy of lexical choice) to measure lexical translation accuracy per source word.
• Show that KD improves global AoLC but harms low-frequency word AoLC.
• Introduce two bilingual data-dependent priors from raw data: Word Alignment Distribution (WAD) and Self-Distilled Distribution (SDD).
• Incorporate priors via a KL divergence term L_prior and combine with NAT loss using a decaying imitation rate lambda.
• Train with both raw and distilled data, evaluating on MaskPredict and Levenshtein Transformer across En-De, Ro-En, Zh-En, Ja-En.
• Demonstrate that combining WAD and SDD yields consistent BLEU and AoLC gains, especially for low-frequency words.

Experimental results

Research questions

• RQ1 Does KD from AT teachers propagate low-frequency lexical errors into NAT models?
• RQ2 Can exposing NAT models to raw data via data-dependent priors recover lost low-frequency lexical information?
• RQ3 Do bilingual priors (WAD and SDD) improve lexical choice and overall translation quality across multiple language pairs and NAT architectures?
• RQ4 How do these priors affect AoLC and low-frequency token recall, and do they preserve or improve BLEU?

Key findings

• KD improves global lexical accuracy (AoLC) but can degrade low-frequency word AoLC as teacher quality improves.
• NAT models trained with raw-data priors show higher AoLC and BLEU across En-De, Zh-En, Ja-En.
• Combining Word Alignment Distribution and Self-Distilled Distribution yields the largest gains (AoLC and BLEU) over KD baselines.
• Low-frequency token accuracy improves significantly (Avg +3.2 AoLC for low-frequency tokens) and more low-frequency tokens are recalled in translations.
• Human evaluation shows reduced lexical choice errors and improved handling of low-frequency words.
• The proposed priors are robust to weaker aligners and alignment noise, still outperforming KD baselines.