[论文解读] The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Training

Andrea Montanari, Yiqiao Zhong|arXiv (Cornell University)|Jul 25, 2020

Neural Networks and Applications被引用 6

一句话总结

本文分析了在懒惰训练下两层神经网络的插值相变现象，表明当隐藏神经元数与输入维度的乘积（Nd）远大于样本量（n）时，神经正切核（NTK）保持可逆，泛化误差可由带有自诱导正则化项的核岭回归良好近似。关键结果是，过参数化使得网络在保持低测试误差的同时，能够对任意标签实现精确插值，即使标签是随机的，这归因于NT区间的隐式偏差。

ABSTRACT

Modern neural networks are often operated in a strongly overparametrized regime: they comprise so many parameters that they can interpolate the training set, even if actual labels are replaced by purely random ones. Despite this, they achieve good prediction error on unseen data: interpolating the training set does not lead to a large generalization error. Further, overparametrization appears to be beneficial in that it simplifies the optimization landscape. Here we study these phenomena in the context of two-layers neural networks in the neural tangent (NT) regime. We consider a simple data model, with isotropic covariates vectors in $d$ dimensions, and $N$ hidden neurons. We assume that both the sample size $n$ and the dimension $d$ are large, and they are polynomially related. Our first main result is a characterization of the eigenstructure of the empirical NT kernel in the overparametrized regime $Nd\gg n$. This characterization implies as a corollary that the minimum eigenvalue of the empirical NT kernel is bounded away from zero as soon as $Nd\gg n$, and therefore the network can exactly interpolate arbitrary labels in the same regime. Our second main result is a characterization of the generalization error of NT ridge regression including, as a special case, min-$\ell_2$ norm interpolation. We prove that, as soon as $Nd\gg n$, the test error is well approximated by the one of kernel ridge regression with respect to the infinite-width kernel. The latter is in turn well approximated by the error of polynomial ridge regression, whereby the regularization parameter is increased by a `self-induced' term related to the high-degree components of the activation function. The polynomial degree depends on the sample size and the dimension (in particular on $\log n/\log d$).

研究动机与目标

• 理解过参数化神经网络在何种条件下能够精确插值训练数据。
• 刻画过参数化状态下神经正切岭回归的泛化误差。
• 解释为何在记忆训练标签的情况下插值不会导致泛化性能下降。
• 建立神经正切模型与高维设置下多项式岭回归之间的联系。

提出的方法

• 分析高维空间中具有各向同性协变量的两层神经网络在神经正切（NT）区间的特性。
• 在 Nd ≫ n 条件下推导经验NT核的特征结构，表明最小特征值远离零。
• 利用浓度不等式和算子范数界，建立NT核对无限宽度极限的稳定性和逼近性。
• 将NT岭回归的泛化误差与具有有效正则化参数的多项式岭回归的泛化误差相联系，该参数额外包含与激活函数高阶分量相关的项。
• 应用Bernstein不等式和基于体积的打包论证，界定网络权重的覆盖数并建立泛化界。
• 采用渐近分析，令 n, d → ∞ 且 n 与 d 多项式相关，推导测试误差的精确近似。

实验结果

研究问题

• RQ1 在何种条件下，具有 N 个隐藏单元的两层神经网络能够插值 n 个训练数据点？
• RQ2 当网络过参数化（Nd ≫ n）且插值任意标签时，泛化误差如何表现？
• RQ3 神经正切岭回归的泛化误差与无限宽度核的核岭回归之间存在何种关系？
• RQ4 懒惰训练区间的隐式正则化如何影响插值区间的泛化性能？
• RQ5 泛化误差能否由多项式岭回归近似？若能，有效正则化参数是什么？

主要发现

• 当 Nd ≫ n 时，经验NT核的最小特征值远离零，确保对任意标签的精确插值。
• NT岭回归的泛化误差可由无限宽度核的核岭回归良好近似。
• 等价多项式岭回归模型中的有效正则化参数包含一个与激活函数高阶分量成正比的自诱导项。
• 等价回归模型中的多项式次数按 log n / log d 的速率增长，反映了样本量与维度之间的相互作用。
• 即使标签是随机的，测试误差仍保持较低，这是由于NT区间的隐式偏差以及过参数化状态下核的稳定性。
• 在 log n = o(min{N, d}) 条件下，泛化误差以高概率有界，表明对过参数化的鲁棒性。