[Paper Review] Self-Distillation Amplifies Regularization in Hilbert Space

Hossein Mobahi, Mehrdad Farajtabar+1 more|arXiv (Cornell University)|Feb 13, 2020

Gaussian Processes and Bayesian Inference41 references98 citations

TL;DR

The paper provides a theoretical analysis of self-distillation in Hilbert space regression with L2 regularization, showing that repeated distillation sparsifies the solution basis and can shift from reduced overfitting to possible underfitting.

ABSTRACT

Knowledge distillation introduced in the deep learning context is a method to transfer knowledge from one architecture to another. In particular, when the architectures are identical, this is called self-distillation. The idea is to feed in predictions of the trained model as new target values for retraining (and iterate this loop possibly a few times). It has been empirically observed that the self-distilled model often achieves higher accuracy on held out data. Why this happens, however, has been a mystery: the self-distillation dynamics does not receive any new information about the task and solely evolves by looping over training. To the best of our knowledge, there is no rigorous understanding of this phenomenon. This work provides the first theoretical analysis of self-distillation. We focus on fitting a nonlinear function to training data, where the model space is Hilbert space and fitting is subject to $\ell_2$ regularization in this function space. We show that self-distillation iterations modify regularization by progressively limiting the number of basis functions that can be used to represent the solution. This implies (as we also verify empirically) that while a few rounds of self-distillation may reduce over-fitting, further rounds may lead to under-fitting and thus worse performance.

Motivation & Objective

• Motivate understanding of why self-distillation improves generalization despite no new task information.
• Formalize self-distillation as an iterative regularization mechanism within a Hilbert space regression setting.
• Characterize how self-distillation alters the regularization and the effective basis used to represent the solution.
• Provide bounds and insights into when self-distillation helps versus when it causes under-fitting.

Proposed method

• Set up a constrained regularized regression problem with a Mercer kernel-based regularizer R(f).
• Derive the KKT conditions leading to a representer-like closed form for the solution f*, using a Green’s function of the kernel operator.
• Express self-distillation as a recurrence on the training label vector y_t via y_t = V^T A_{t-1} V y_{t-1}, where A_t are diagonal matrices.
• Show that f* in each round has a closed form involving (c I + G)^{-1} with G a Gram-like matrix built from the kernel.
• Analyze the evolution of the product B_t = ∏_{i=0}^t A_i to reveal progressive sparsification of basis weights.
• Discuss the near-interpolation regime and contrast with early stopping in terms of sparsity and regularization.

Experimental results

Research questions

• RQ1 Does self-distillation act as a regularizer in Hilbert space regression, and if so, how does it modify the effective basis?
• RQ2 How does the number of self-distillation rounds affect generalization versus underfitting?
• RQ3 Can we bound the number of nonzero basis components and describe the sparsity pattern induced by self-distillation?
• RQ4 How do the dynamics of self-distillation relate to the interpolation regime and potential generalization benefits?
• RQ5 What extensions to multiclass settings and generalization bounds arise from this analysis?

Key findings

• Self-distillation iterates modify regularization, progressively limiting the number of basis functions used to represent the solution.
• The product of diagonal matrices in the distillation steps, B_t, becomes increasingly sparse, reducing effective model capacity.
• There exists a guaranteed number of rounds before the solution collapses to the zero function (under a fixed error tolerance).
• Near-interpolation regimes (small epsilon) enhance sparsity, suggesting deeper distillation can increase regularization if kept before collapse.
• Early stopping via larger training error is not equivalent to self-distillation sparsification and behaves differently in terms of regularization.