[Paper Review] Multilingual Neural Machine Translation with Knowledge Distillation

Xu Tan, Yi Ren+4 more|arXiv (Cornell University)|Feb 27, 2019

Natural Language Processing Techniques129 citations

TL;DR

The paper introduces a multilingual NMT framework that uses knowledge distillation from individually trained language-pair models to train a single multilingual model, achieving accuracy close to or surpassing individual models with far fewer parameters.

ABSTRACT

Multilingual machine translation, which translates multiple languages with a single model, has attracted much attention due to its efficiency of offline training and online serving. However, traditional multilingual translation usually yields inferior accuracy compared with the counterpart using individual models for each language pair, due to language diversity and model capacity limitations. In this paper, we propose a distillation-based approach to boost the accuracy of multilingual machine translation. Specifically, individual models are first trained and regarded as teachers, and then the multilingual model is trained to fit the training data and match the outputs of individual models simultaneously through knowledge distillation. Experiments on IWSLT, WMT and Ted talk translation datasets demonstrate the effectiveness of our method. Particularly, we show that one model is enough to handle multiple languages (up to 44 languages in our experiment), with comparable or even better accuracy than individual models.

Motivation & Objective

• Motivate multilingual NMT to reduce training and serving costs while maintaining high accuracy across many language pairs.
• Leverage knowledge distillation from language-pair teacher models to train a single multilingual student model.
• Demonstrate that a multilingual model can achieve comparable or better accuracy than separate models on diverse datasets (IWSLT, WMT, Ted talk).
• Explore selective distillation and Top-K distillation to manage training efficiency and memory usage.

Proposed method

• Train individual language-pair models as teachers for each language pair.
• Train a single multilingual student model to minimize both ground-truth NLL and distillation loss against all teachers.
• Use selective distillation to switch off distillation for language pairs once the student surpasses the teacher by a threshold.
• Optionally use Top-K distillation to reduce memory by distilling only the top-K teacher probabilities per token.
• During training, progressively compare student and teacher accuracies and adjust distillation usage accordingly.

Experimental results

Research questions

• RQ1 Can a multilingual NMT model reach the accuracy of language-pair specialized models using knowledge distillation from those models?
• RQ2 Does selective distillation help maintain or improve performance while reducing negative impacts from weak teachers?
• RQ3 What are the memory and performance trade-offs when using Top-K distillation in a large multilingual setting?

Key findings

• The multilingual baseline is typically worse than individual models, but the proposed distillation method closes or surpasses the gap for many language pairs.
• Multi-Distillation achieves similar or better BLEU than individual models for most languages with only 1/44 the total number of parameters in the Ted Talk setting.
• Selective distillation improves BLEU on the majority of languages compared to distilling from all teachers constantly.
• Top-K distillation achieves comparable or better performance than full distribution distillation while reducing memory usage.
• Back distillation can improve some individual language models after multilingual distillation, though results vary by language.