[论文解读] The generalization error of random features regression: Precise asymptotics and double descent curve

Mei Song, Andrea Montanari|arXiv (Cornell University)|Aug 14, 2019

Random Matrices and Applications参考文献 31被引用 248

一句话总结

本文推导了随机特征岭回归的精确高维渐近，显示一个 double-descent 泛化曲线，并且在高度过参数化的区间中、有无正则化，最优测试误差都会出现。

ABSTRACT

Deep learning methods operate in regimes that defy the traditional statistical mindset. Neural network architectures often contain more parameters than training samples, and are so rich that they can interpolate the observed labels, even if the latter are replaced by pure noise. Despite their huge complexity, the same architectures achieve small generalization error on real data. This phenomenon has been rationalized in terms of a so-called `double descent' curve. As the model complexity increases, the test error follows the usual U-shaped curve at the beginning, first decreasing and then peaking around the interpolation threshold (when the model achieves vanishing training error). However, it descends again as model complexity exceeds this threshold. The global minimum of the test error is found above the interpolation threshold, often in the extreme overparametrization regime in which the number of parameters is much larger than the number of samples. Far from being a peculiar property of deep neural networks, elements of this behavior have been demonstrated in much simpler settings, including linear regression with random covariates. In this paper we consider the problem of learning an unknown function over the $d$-dimensional sphere $\mathbb S^{d-1}$, from $n$ i.i.d. samples $(\boldsymbol x_i, y_i)\in \mathbb S^{d-1} imes \mathbb R$, $i\le n$. We perform ridge regression on $N$ random features of the form $σ(\boldsymbol w_a^{\mathsf T} \boldsymbol x)$, $a\le N$. This can be equivalently described as a two-layers neural network with random first-layer weights. We compute the precise asymptotics of the test error, in the limit $N,n,d o \infty$ with $N/d$ and $n/d$ fixed. This provides the first analytically tractable model that captures all the features of the double descent phenomenon without assuming ad hoc misspecification structures.

研究动机与目标

• 在一个非平凡的非参数设置的随机特征回归中，动员并分析 double descent 现象。
• 在 N/d 与 n/d 固定的成比例（proportional）区间中推导测试误差的精确渐近。
• 表征正则化和信噪比如何影响泛化以及插值阈值的位置。

提出的方法

• 将学习问题建模为对 N 个带激活 σ 的随机特征进行的 ridge regression，在 d 维球面上取样的 n 个样本上训练。
• 在 N,n,d -> ∞ 且 N/d -> psi1、n/d -> psi2 的极限下推导测试误差 R_RF 的精确渐近。
• 通过块结构随机矩阵的 Stieltjes 变换，将预测误差表示为 psi1、psi2、lambda 与数据统计量的函数。
• 在渐近极限下证明 random features 与 Gaussian covariates 模型之间的等价性，以获得直觉。
• 提供特殊情形的简化，包括 ridgeless 极限和高度过参数化的情形。
• 将结果与 kernel 视角相关联，并讨论自诱导正则化机制。

实验结果

研究问题

• RQ1 在高维比例极限下，随机特征 ridge 回归的精确渐近预测误差是多少？
• RQ2 模型复杂度（N/d 和 n/d）与正则化（lambda）如何在这个非参数设定中相互作用以产生 double descent？
• RQ3 在高度过参数化的区间内，何种条件下 random features 模型表现出最优泛化？
• RQ4 Gaussian covariates 代理能否再现与 random features 相同的渐近泛化行为？
• RQ5 线性目标函数与非线性目标函数如何影响测试误差的渐近性质？

主要发现

• 本文在成比例 regime 中获得测试误差的精确渐近，捕捉了 double descent 现象的所有特征。
• 在超过临界信噪比时，最小测试误差由极度过参数化的插值器实现，训练误差趋近于零。
• 正则化的效果取决于 SNR，可能有帮助也可能有害，在某个临界 SNR 处发现相变，此时最优 lambda 会发生转变。
• ridgeless 极限 (lambda -> 0) 常常在高度过参数化区间产生近似插值器，在统计意义上是最优的。
• 分析显示方差和偏差都可在插值阈值处达到峰值，且 double descent 即使在无噪声设置下也会持续。
• 该模型表明在不需要特定错误设定假设的情况下也能实现最优泛化，并且在合适条件下强烈的过参数化是有益的。