[Paper Review] Ensemble Distillation for Neural Machine Translation

Markus Freitag, Yaser Al-Onaizan+1 more|arXiv (Cornell University)|Feb 6, 2017

Natural Language Processing Techniques8 references64 citations

TL;DR

The paper demonstrates knowledge distillation for NMT to transfer ensemble and oracle Bleu teacher quality into a single student model, including data filtering and training with same-architecture teachers, achieving faster decoding with comparable or better translation quality. It also shows data pruning and model-size reduction without loss of performance.

ABSTRACT

Knowledge distillation describes a method for training a student network to perform better by learning from a stronger teacher network. Translating a sentence with an Neural Machine Translation (NMT) engine is time expensive and having a smaller model speeds up this process. We demonstrate how to transfer the translation quality of an ensemble and an oracle BLEU teacher network into a single NMT system. Further, we present translation improvements from a teacher network that has the same architecture and dimensions of the student network. As the training of the student model is still expensive, we introduce a data filtering method based on the knowledge of the teacher model that not only speeds up the training, but also leads to better translation quality. Our techniques need no code change and can be easily reproduced with any NMT architecture to speed up the decoding process.

Motivation & Objective

• Motivate the use of knowledge distillation to improve NMT quality while enabling faster decoding.
• Demonstrate transferring ensemble/oracle Bleu teacher knowledge into a single student model, including cases where teacher and student share architecture and size.
• Show data filtering techniques based on teacher predictions to speed up and improve training.
• Evaluate effects of teacher types, data filtering, and model-size reduction on translation quality for German-English.

Proposed method

• Translate full training data with the teacher to create pseudo-targets for the student.
• Train the student on (source, teacher-translation) pairs, optionally combining with original references.
• Experiment with different teacher types: single same-architecture teacher, ensemble of 6 models, and oracle Bleu-based teacher.
• Apply Ter-based data pruning to remove noisy sentence pairs.
• Reduce student model size by lowering embedding and hidden-layer dimensions while preserving or improving performance.

Experimental results

Research questions

• RQ1 Does knowledge distillation from an ensemble teacher match or exceed single-teacher distillation in NMT?
• RQ2 Can a student with the same architecture as the teacher still gain from distillation, and by how much?
• RQ3 Does data filtering based on translation quality (Ter) improve training efficiency and translation quality?
• RQ4 How does oracle Bleu-based distillation compare to ensemble-based distillation in NMT performance?
• RQ5 What is the impact of reducing student model size on translation quality when guided by an ensemble teacher?

Key findings

• Ensemble teacher distillation improves Bleu by about 2.0 points and Ter by about 2.2 points over baselines with data pruning (Ter ≤ 0.8).
• Single-architecture distillation with forward translations plus original references yields ~1.4 Bleu and ~1.0–1.2 Ter gains.
• Oracle Bleu teacher distillation provides gains but is slightly below ensemble-teacher results.
• Pruning noisy training data (Ter > 0.8) reduces training data by ~12% and speeds training while maintaining or improving quality.
• Student models with reduced embedding/hidden sizes (e.g., 150/300) can match or surpass the baseline and even beat larger teachers in Bleu/Ter under distillation.
• Knowledge distillation from an ensemble teacher enables a smaller student to achieve similar or better translation quality with substantially faster decoding and smaller memory footprint.