[论文解读] Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime

Stéphane d’Ascoli, Maria Refinetti+2 more|arXiv (Cornell University)|Mar 2, 2020

Stochastic Gradient Optimization Techniques被引用 57

一句话总结

该论文在 random features 的作用下分析 lazy learning regime 中的 double descent，推导出测试误差的精确偏差-方差分解，并展示集成和过参数化如何抑制插值阈值处的过拟合峰值。

ABSTRACT

Deep neural networks can achieve remarkable generalization performances while interpolating the training data perfectly. Rather than the U-curve emblematic of the bias-variance trade-off, their test error often follows a "double descent" - a mark of the beneficial role of overparametrization. In this work, we develop a quantitative theory for this phenomenon in the so-called lazy learning regime of neural networks, by considering the problem of learning a high-dimensional function with random features regression. We obtain a precise asymptotic expression for the bias-variance decomposition of the test error, and show that the bias displays a phase transition at the interpolation threshold, beyond which it remains constant. We disentangle the variances stemming from the sampling of the dataset, from the additive noise corrupting the labels, and from the initialization of the weights. Following up on Geiger et al. 2019, we first show that the latter two contributions are the crux of the double descent: they lead to the overfitting peak at the interpolation threshold and to the decay of the test error upon overparametrization. We then quantify how they are suppressed by ensemble averaging the outputs of K independently initialized estimators. When K is sent to infinity, the test error remains constant beyond the interpolation threshold. We further compare the effects of overparametrizing, ensembling and regularizing. Finally, we present numerical experiments on classic deep learning setups to show that our results hold qualitatively in realistic lazy learning scenarios.

研究动机与目标

• 理解神经网络惰性 regime 中驱动 double descent 的机制。
• 辨析测试误差来自噪声、初始化和采样方差的贡献。
• 给出集成如何影响这些方差的精确渐近公式。
• 比较过参数化、集成和正则化对泛化性能的影响。

提出的方法

• 将神经网络建模为 Random Features：固定随机的一层权重、通过岭回归训练二层权重。
• 推导测试误差的偏差-方差分解，包括噪声、初始化、采样和偏差项。
• 在高维极限下使用 replica 方法推导这些项的精确渐近表达。
• 通过对 K 个独立初始化的估计器的输出取平均来分析集成的效应，并推导其对测试误差的影响。
• 将 RF 结果与在 P→∞ 极限下的核岭回归联系起来，并与经验深度学习场景进行比较。

实验结果

研究问题

• RQ1 在惰性学习中，贡献于测试误差的不同方差和偏差的来源有哪些，它们在插值阈值附近的行为如何？
• RQ2 集成如何影响不同方差分量以及整体的 double-descent 曲线？
• RQ3 过参数化、集成和正则化在缓解过拟合峰值方面的比较。
• RQ4 RF/核结果在真实的惰性学习神经网络与数据中的适用程度？

主要发现

• 测试误差分解为噪声、初始化、采样方差和偏差，贝叶斯误差作为残差项。
• 插值阈值导致噪声方差和初始化方差发散，而采样方差与偏差呈现拐点和平台期，均被正则化平滑。
• 在插值阈值之上，偏差和采样方差基本保持不变，过参数化的好处来自降低噪声方差和初始化方差。
• 对受影响的方差项，集成 K 个独立初始化的估计器将发散减小一个因子 1/K，当 K→∞ 时保持恒定的测试误差。
• 过参数化和集成在抑制 double-descent 峰值方面具有相似的定性效果，解析表达式量化了它们的相对影响。
• 有限大小的仿真验证了渐近预测，惰性状态下的 CNN/DNN 实验与 RF 结果在定性上一致。