[论文解读] A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning

Alicia Curth, Alan Jeffares+1 more|arXiv (Cornell University)|Oct 29, 2023

Neural Networks and Applications被引用 7

一句话总结

本文通过表明测试误差中看似存在的第二次下降实际上是多重复杂度轴混淆的产物，挑战了经典机器学习模型中双下降现象的解释。通过引入基于非参数平滑的有效参数计数，作者证明当复杂度被正确衡量时，测试误差曲线会恢复为传统的U形，从而解决了双下降与经典统计理论之间的矛盾。

ABSTRACT

Conventional statistical wisdom established a well-understood relationship between model complexity and prediction error, typically presented as a U-shaped curve reflecting a transition between under- and overfitting regimes. However, motivated by the success of overparametrized neural networks, recent influential work has suggested this theory to be generally incomplete, introducing an additional regime that exhibits a second descent in test error as the parameter count p grows past sample size n - a phenomenon dubbed double descent. While most attention has naturally been given to the deep-learning setting, double descent was shown to emerge more generally across non-neural models: known cases include linear regression, trees, and boosting. In this work, we take a closer look at evidence surrounding these more classical statistical machine learning methods and challenge the claim that observed cases of double descent truly extend the limits of a traditional U-shaped complexity-generalization curve therein. We show that once careful consideration is given to what is being plotted on the x-axes of their double descent plots, it becomes apparent that there are implicitly multiple complexity axes along which the parameter count grows. We demonstrate that the second descent appears exactly (and only) when and where the transition between these underlying axes occurs, and that its location is thus not inherently tied to the interpolation threshold p=n. We then gain further insight by adopting a classical nonparametric statistics perspective. We interpret the investigated methods as smoothers and propose a generalized measure for the effective number of parameters they use on unseen examples, using which we find that their apparent double descent curves indeed fold back into more traditional convex shapes - providing a resolution to tensions between double descent and statistical intuition.

研究动机与目标

• 挑战非神经网络模型（如决策树、提升模型和线性回归）中双下降现象否定经典U形泛化曲线的说法。
• 识别出双下降图隐式地结合了多个复杂度轴，当沿单一复合轴绘制时会导致误导性解释。
• 为平滑器提出一种广义的有效参数计数，以衡量未见数据上的模型复杂度。
• 证明双下降并非源于插值阈值 $p = n$ 本身，而是轴转换的产物。
• 通过以有效自由度重新表达泛化误差，将双下降与经典统计直觉统一起来。

提出的方法

• 从非参数视角出发，将决策树、提升模型和线性模型解释为平滑器。
• 基于测试输入对预测的影响，利用平滑矩阵的迹，定义有效参数数量 $p^{ ext{test}}_{ ilde{ extbf{s}}}$。
• 在泛化图中使用此有效参数计数作为x轴，而非原始模型参数或超参数。
• 通过沿独立超参数轴（如树深度和估计器数量）分解实验，分析树和提升模型中的双下降现象。
• 使用 $p^{ ext{test}}_{ ilde{ extbf{s}}}$ 重构泛化曲线，表明第二次下降消失，曲线变为凸形。
• 将相同框架应用于具有最小范数解的线性回归，将其与无监督降维及有效自由度联系起来。

实验结果

研究问题

• RQ1 在非深度学习模型中观察到的双下降是否真的偏离了经典U形泛化曲线？
• RQ2 当参数数量超过样本量时，双下降图中第二次下降的原因是什么？
• RQ3 双下降是否可被解释为多个复杂度轴合并的产物，而非模型行为的根本性转变？
• RQ4 通过有效参数计数衡量复杂度，是否能恢复这些模型中的经典U形曲线？
• RQ5 有效参数计数在不同模型类别中与插值阈值 $p = n$ 的关系如何？

主要发现

• 树和提升模型中的双下降源于依次增加多个超参数（如树深度和估计器数量），而非单一复杂度轴。
• 当沿独立轴绘制时，树深度和估计器数量各自均表现出经典的U形曲线，表明并未根本偏离经典理论。
• 双下降图中的第二次下降恰好出现在这些底层复杂度轴的转换点，而非固有地出现在 $p = n$ 处。
• 对于具有最小范数解的线性回归，有效参数计数 $p^{ ext{test}}_{ ilde{ extbf{s}}}$ 保持有界，即使原始参数计数增长，也不会超过插值阈值。
• 当测试误差以 $p^{ ext{test}}_{ ilde{ extbf{s}}}$ 为横轴绘制时，所有测试模型（包括树、提升模型和RFF回归）的双下降曲线均坍缩为凸形U曲线。
• 表现最佳的插值模型（训练误差为零）始终是 $p^{ ext{test}}_{ ilde{ extbf{s}}}$ 最低的模型，证实了有效参数计数的预测能力。