[Paper Review] Understanding Knowledge Distillation in Non-autoregressive Machine Translation

Chunting Zhou, Graham Neubig+1 more|arXiv (Cornell University)|Nov 6, 2019

Natural Language Processing Techniques33 references117 citations

TL;DR

The paper investigates why sequence-level knowledge distillation helps non-autoregressive translation (NAT), introduces data complexity and faithfulness metrics, analyzes teacher-student dynamics across multiple models, and proposes methods to tailor distilled data to NAT capacity, achieving state-of-the-art NAT results close to autoregressive baselines on WMT14 En-De.

ABSTRACT

Non-autoregressive machine translation (NAT) systems predict a sequence of output tokens in parallel, achieving substantial improvements in generation speed compared to autoregressive models. Existing NAT models usually rely on the technique of knowledge distillation, which creates the training data from a pretrained autoregressive model for better performance. Knowledge distillation is empirically useful, leading to large gains in accuracy for NAT models, but the reason for this success has, as of yet, been unclear. In this paper, we first design systematic experiments to investigate why knowledge distillation is crucial to NAT training. We find that knowledge distillation can reduce the complexity of data sets and help NAT to model the variations in the output data. Furthermore, a strong correlation is observed between the capacity of an NAT model and the optimal complexity of the distilled data for the best translation quality. Based on these findings, we further propose several approaches that can alter the complexity of data sets to improve the performance of NAT models. We achieve the state-of-the-art performance for the NAT-based models, and close the gap with the autoregressive baseline on WMT14 En-De benchmark.

Motivation & Objective

• Investigate why knowledge distillation improves NAT training and performance.
• Quantify how distilled data affects data complexity and translation modes.
• Explore the relationship between NAT model capacity and the optimal distilled data complexity.
• Propose methods to tailor distilled data to NAT capacity to close the gap with autoregressive models.

Proposed method

• Use synthetic experiments to visualize mode reduction due to distillation.
• Define and compute data-level metrics: conditional entropy (complexity) and KL-divergence-based faithfulness (F(d)).
• Systematically evaluate four AT teachers and six NAT students on WMT14 En-De with real and distilled targets.
• Analyze decoding strategies (sampling, top-k sampling, greedy, beam search) and their impact on distillation quality.
• Experiment with data-level improvements (Born-Again Networks, Mixture-of-Experts, sequence-level interpolation) to adjust distilled data complexity and fidelity.
• Report correlations between NAT capacity, distilled data complexity, and translation quality across models.

Experimental results

Research questions

• RQ1 How does knowledge distillation reduce multi-modality in NAT outputs and why does this help NAT training?
• RQ2 What is the relationship between NAT model capacity and the optimal complexity of distilled training data?
• RQ3 Can altering the distilled data (via decoding method, BANs, MoE, interpolation) further close the performance gap between NAT and autoregressive models?

Key findings

• Distillation reduces output modes and complexity, and higher-capacity NAT models require more complex distilled data for best performance.
• Higher-capacity teachers yield distilled data with greater complexity and better faithfulness to real data, boosting NAT BLEU.
• Beam search distillation reduces data complexity most effectively while maintaining faithfulness, improving NAT performance.
• Weaker NAT models benefit from distilled data with simpler complexity, while stronger NAT models benefit from more complex distilled data.
• Born-Again Networks and mixture-of-experts can tailor distilled data to reduce complexity or improve faithfulness, improving NAT results.
• Simple data-level interpolation (selecting high-BLEU beam-hypotheses) can further enhance NAT performance for certain models.