[论文解读] Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation

Kenneth Borup, Lars Nørvang Andersen|arXiv (Cornell University)|Feb 25, 2021

Adversarial Robustness in Machine Learning参考文献 39被引用 5

一句话总结

本文首次对核回归中的自蒸馏进行了理论分析，表明在每个蒸馏步骤中引入真实标签可有效增强正则化。论文推导出最优加权参数的闭式解，并证明无限蒸馏步骤的极限收敛于正则化程度更高的核岭回归问题，从而解释了为何自蒸馏能提升泛化性能，而不仅仅是知识迁移。

ABSTRACT

Knowledge distillation is classically a procedure where a neural network is trained on the output of another network along with the original targets in order to transfer knowledge between the architectures. The special case of self-distillation, where the network architectures are identical, has been observed to improve generalization accuracy. In this paper, we consider an iterative variant of self-distillation in a kernel regression setting, in which successive steps incorporate both model outputs and the ground-truth targets. This allows us to provide the first theoretical results on the importance of using the weighted ground-truth targets in self-distillation. Our focus is on fitting nonlinear functions to training data with a weighted mean square error objective function suitable for distillation, subject to $\ell_2$ regularization of the model parameters. We show that any such function obtained with self-distillation can be calculated directly as a function of the initial fit, and that infinite distillation steps yields the same optimization problem as the original with amplified regularization. Furthermore, we provide a closed form solution for the optimal choice of weighting parameter at each step, and show how to efficiently estimate this weighting parameter for deep learning and significantly reduce the computational requirements compared to a grid search.

研究动机与目标

• 为了从理论上解释为何自蒸馏在无新数据的情况下仍能提升泛化性能。
• 为了分析在 $μat$-正则化下，真实标签在迭代自蒸馏中的作用。
• 为了推导每个蒸馏步骤中最优加权参数 $α$ 的闭式解。
• 为了刻画无限蒸馏步骤在核回归框架下的极限行为。
• 为了证明自蒸馏等价于求解正则化参数被增强的核岭回归问题。

提出的方法

• 将自蒸馏建模为一个带加权均方误差目标的迭代优化问题，结合真实标签与模型预测标签。
• 使用带有 $μat$-正则化的核岭回归，以在过参数化状态下建模深度学习的泛化行为。
• 应用Karush-Kuhn-Tucker（KKT）条件，推导每个蒸馏步骤的最优解。
• 利用格林函数和核矩阵的谱分解，推导出最优加权参数 $α$ 的闭式表达式。
• 建立步骤 $τ$ 的解仅依赖于初始拟合和加权参数序列的结论。
• 证明无限蒸馏步骤的极限等价于求解原始问题，但正则化参数被增强。

实验结果

研究问题

• RQ1 在迭代自蒸馏中，真实标签的引入如何影响正则化强度？
• RQ2 在每个蒸馏步骤中，真实标签与模型预测标签之间的最优加权比例如何确定？
• RQ3 无限自蒸馏是否收敛到一个定义良好的优化问题？
• RQ4 迭代自蒸馏过程能否被简化为闭式解？
• RQ5 自蒸馏与正则化参数被增强的核岭回归之间有何关系？

主要发现

• 通过自蒸馏获得的任意模型均可直接从初始模型拟合中计算得出，无需迭代微调。
• 无限蒸馏步骤收敛于正则化参数被 $\frac{1}{1-\alpha}$ 缩放的核岭回归问题的同一解。
• 每个步骤的最优加权参数 $\alpha$ 具有基于核矩阵和目标范数的闭式表达式。
• 该方法可实现 $\alpha$ 的高效估计，无需网格搜索，显著降低计算成本。
• 理论框架解释了为何自蒸馏能提升泛化性能：其有效增强了正则化，从而在过参数化模型中减少过拟合。
• 每个步骤 $\tau$ 的解可表示为 $\mathbf{y}^{(\tau)} = \mathbf{g}(\mathbf{x})^\top (\mathbf{G} + \lambda_\tau \mathbf{I})^{-1} (\alpha \mathbf{y} + (1-\alpha)\mathbf{y}^{(\tau-1)})$，将蒸馏与核方法联系起来。