[Paper Review] Sequence-Level Knowledge Distillation

Yoon Kim, Alexander M. Rush|arXiv (Cornell University)|Jun 25, 2016

Natural Language Processing Techniques47 references117 citations

TL;DR

The paper adapts knowledge distillation to neural machine translation, introducing sequence-level variants that outperform word-level distillation and enable greedy decoding, faster inference, and substantial compression through pruning.

ABSTRACT

Neural machine translation (NMT) offers a novel alternative formulation of translation that is potentially simpler than statistical approaches. However to reach competitive performance, NMT models need to be exceedingly large. In this paper we consider applying knowledge distillation approaches (Bucila et al., 2006; Hinton et al., 2015) that have proven successful for reducing the size of neural models in other domains to the problem of NMT. We demonstrate that standard knowledge distillation applied to word-level prediction can be effective for NMT, and also introduce two novel sequence-level versions of knowledge distillation that further improve performance, and somewhat surprisingly, seem to eliminate the need for beam search (even when applied on the original teacher model). Our best student model runs 10 times faster than its state-of-the-art teacher with little loss in performance. It is also significantly better than a baseline model trained without knowledge distillation: by 4.2/1.7 BLEU with greedy decoding/beam search. Applying weight pruning on top of knowledge distillation results in a student model that has 13 times fewer parameters than the original teacher model, with a decrease of 0.4 BLEU.

Motivation & Objective

• Reduce NMT model size without sacrificing performance.
• Investigate word-level and sequence-level knowledge distillation for NMT.
• Explore sequence-level interpolation to leverage teacher distributions.
• Evaluate decoding speed and compression when using distilled and pruned models.

Proposed method

• Apply standard word-level knowledge distillation for NMT by matching teacher/student word distributions.
• Propose sequence-level knowledge distillation (Seq-KD) by training the student on the teacher’s beam-searched output (mode of q(t|s)).
• Introduce sequence-level interpolation (Seq-Inter) to mix teacher-generated data with original data.
• Use beam search (K=5) to approximate the teacher distribution and evaluate on English–German and Thai–English tasks.
• Perform fine-tuning and data generation for efficiency (e.g., Seq-Inter on 50% of data for English–German).
• Explore weight pruning to further reduce parameters after distillation.

Experimental results

Research questions

• RQ1 Can sequence-level knowledge distillation outperform standard word-level distillation in NMT?
• RQ2 Does Seq-KD enable greedy decoding to approach or match beam-search quality?

Key findings

• Seq-KD yields substantial BLEU gains over baselines, surpassing Word-KD on English→German and performing similarly on Thai→English.
• Combining Seq-KD with Word-KD provides orthogonal gains, especially for smaller student models (e.g., 2×300/2×100).
• Seq-Inter can match or exceed teacher-beam performance with greedy decoding, enabling faster inference.
• Greedy decoding on Seq-KD models can be roughly 10× faster than beam search on the larger teacher with comparable BLEU.
• Weight pruning can yield up to 13× fewer parameters with only small BLEU loss (e.g., 0.4 BLEU drop at 80% prune).
• Seq-KD tends to produce a peaked distribution around the teacher’s mode, aiding effective greedy decoding and reducing the need for beam search.