[论文解读] Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning

Zeyuan Allen-Zhu, Yuanzhi Li|arXiv (Cornell University)|Dec 17, 2020

Adversarial Robustness in Machine Learning参考文献 92被引用 151

一句话总结

该论文展示了理论与经验研究，表明在同一数据上训练、架构相同的神经网络集成在多视图数据结构下可显著提高测试准确率，并且这一提升可以蒸馏为单一模型；还分析自蒸馏作为隐式的集成+蒸馏。

ABSTRACT

We formally study how ensemble of deep learning models can improve test accuracy, and how the superior performance of ensemble can be distilled into a single model using knowledge distillation. We consider the challenging case where the ensemble is simply an average of the outputs of a few independently trained neural networks with the SAME architecture, trained using the SAME algorithm on the SAME data set, and they only differ by the random seeds used in the initialization. We show that ensemble/knowledge distillation in Deep Learning works very differently from traditional learning theory (such as boosting or NTKs, neural tangent kernels). To properly understand them, we develop a theory showing that when data has a structure we refer to as ``multi-view'', then ensemble of independently trained neural networks can provably improve test accuracy, and such superior test accuracy can also be provably distilled into a single model by training a single model to match the output of the ensemble instead of the true label. Our result sheds light on how ensemble works in deep learning in a way that is completely different from traditional theorems, and how the ``dark knowledge'' is hidden in the outputs of the ensemble and can be used in distillation. In the end, we prove that self-distillation can also be viewed as implicitly combining ensemble and knowledge distillation to improve test accuracy.

研究动机与目标

• Explain why ensemble methods improve test accuracy in deep learning beyond traditional learning theory.
• Introduce and formalize a multi-view data setting where ensemble benefits can be proven.
• Show that ensemble improvements can be distilled into a single model trained on the same data.
• Demonstrate that self-distillation effectively combines ensemble and distillation to boost performance.

提出的方法

• Theoretical analysis for a two-layer convolutional network with smoothed ReLU activations.
• Definition of multi-view and single-view data distributions and a corresponding data-generating process.
• Gradient-descent training results showing a single model achieves perfect training accuracy but near-random test error on D.
• Proof that an ensemble of independently trained models achieves substantially better test accuracy.
• Demonstration that training a new model to imitate the ensemble outputs (knowledge distillation) yields similarly improved test accuracy.
• Argument that self-distillation acts as implicit ensemble+distillation and yields further gains.

实验结果

研究问题

• RQ1 How does averaging the outputs of independently trained, identically structured neural networks affect test accuracy under a multi-view data setting?
• RQ2 Can the performance gain of an ensemble be replicated by training a single model to mimic the ensemble outputs (knowledge distillation) on the same training data?
• RQ3 What is the mechanism behind dark knowledge and its role in distillation and self-distillation?
• RQ4 How does self-distillation relate to implicit ensemble and distillation in deep learning?

主要发现

• A single model can achieve perfect training accuracy but only 0.49μ–0.51μ test error under the proposed setup.
• An ensemble of L independently trained models achieves test error ≤ 0.01μ with high probability.
• A separate model trained to match the ensemble output (knowledge distillation) also attains test error ≤ 0.01μ.
• Self-distillation (distilling a model from another model of the same size) can achieve test error ≤ 0.26μ.
• Knowledge distillation over random features (NTK) does not replicate ensemble benefits, highlighting the difference between NTK/per-feature views and real deep learning feature learning.
• The results emphasize that ensemble/knowledge distillation in deep learning emerge from feature-learning dynamics under multi-view data, not simply from traditional ensemble theories.