[论文解读] Teacher-Student Architecture for Knowledge Distillation: A Survey

Chengming Hu, Xuan Li+5 more|arXiv (Cornell University)|Aug 8, 2023

Neural Networks and Applications被引用 11

一句话总结

本综述在多种知识蒸馏目标、知识表示和学习方案下审视教师-学生架构，强调它们在压缩之外的应用并概述未来的研究方向。

ABSTRACT

Although Deep neural networks (DNNs) have shown a strong capacity to solve large-scale problems in many areas, such DNNs are hard to be deployed in real-world systems due to their voluminous parameters. To tackle this issue, Teacher-Student architectures were proposed, where simple student networks with a few parameters can achieve comparable performance to deep teacher networks with many parameters. Recently, Teacher-Student architectures have been effectively and widely embraced on various knowledge distillation (KD) objectives, including knowledge compression, knowledge expansion, knowledge adaptation, and knowledge enhancement. With the help of Teacher-Student architectures, current studies are able to achieve multiple distillation objectives through lightweight and generalized student networks. Different from existing KD surveys that primarily focus on knowledge compression, this survey first explores Teacher-Student architectures across multiple distillation objectives. This survey presents an introduction to various knowledge representations and their corresponding optimization objectives. Additionally, we provide a systematic overview of Teacher-Student architectures with representative learning algorithms and effective distillation schemes. This survey also summarizes recent applications of Teacher-Student architectures across multiple purposes, including classification, recognition, generation, ranking, and regression. Lastly, potential research directions in KD are investigated, focusing on architecture design, knowledge quality, and theoretical studies of regression-based learning, respectively. Through this comprehensive survey, industry practitioners and the academic community can gain valuable insights and guidelines for effectively designing, learning, and applying Teacher-Student architectures on various distillation objectives.

研究动机与目标

• 推动教师-学生架构在模型压缩之外的广泛应用。
• 系统性地对蒸馏目标及知识如何表示与传递进行分类。
• 在教师-学生框架下总结代表性的学习算法和蒸馏方案。
• 突出在分类、识别、生成、排序和回归等领域的应用。
• 指出在架构设计、知识质量与理论方面的开放挑战与方向。

提出的方法

• 定义蒸馏目标的分类法：知识压缩、扩展、适应和增强。
• 详述知识表示：基于响应、中间表示、基于关系，以及基于互信息。
• 综述学习算法和蒸馏方案：多教师、基于图、联邦、跨模态、在线和自蒸馏。
• 讨论结合交叉熵、KL 散度以及基于距离/角度的损失的优化目标。
• 总结应用并给出未来研究方向。

实验结果

研究问题

• RQ1 教师-学生架构在模型压缩之外可以支持哪些蒸馏目标？
• RQ2 不同的知识表示和优化策略在教师-学生蒸馏中如何相互作用？
• RQ3 哪些学习算法和蒸馏方案在不同任务中有效？
• RQ4 在知识蒸馏中的架构设计与知识质量方面有哪些未來的研究方向？
• RQ5 教师-学生蒸馏方法在分类、识别、生成、排序和回归等任务中的应用有多广泛？

主要发现

• 教师-学生架构在压缩之外还能实现包括扩展、适应和增强在内的多种蒸馏目标。
• 讨论了四种知识表示：基于响应、中间、基于关系以及基于互信息。
• 综述了多种学习算法和蒸馏方案，包括多教师、基于图、联邦、跨模态蒸馏，以及在线和自蒸馏。
• 应用覆盖在各领域的分类、识别、生成、排序和回归。
• 本文指出在架构设计、知识质量以及基于回归的学习理论方面的未来方向。