[论文解读] What Causes the Test Error? Going Beyond Bias-Variance via ANOVA

Licong Lin, Edgar Dobriban|arXiv (Cornell University)|Jan 1, 2021

Neural Networks and Applications被引用 4

一句话总结

本文提出一种基于对称方差分析（ANOVA）的分解方法，用于分析两层线性与非线性网络中的测试误差方差，揭示了训练数据与初始化之间的交互作用通常主导方差，超过其各自独立效应。该研究识别出方差行为中的相变现象，并利用哈尓随机矩阵的确定性等价物，建立了方差分量的单峰性与单调性性质。

ABSTRACT

Modern machine learning methods are often overparametrized, allowing adaptation to the data at a fine level. This can seem puzzling; in the worst case, such models do not need to generalize. This puzzle inspired a great amount of work, arguing when overparametrization reduces test error, in a phenomenon called double descent. Recent work aimed to understand in greater depth why overparametrization is helpful for generalization. This leads to discovering the unimodality of variance as a function of the level of parametrization, and to decomposing the variance into that arising from label noise, initialization, and randomness in the training data to understand the sources of the error. In this work we develop a deeper understanding of this area. Specifically, we propose using the analysis of variance (ANOVA) to decompose the variance in the test error in a symmetric way, for studying the generalization performance of certain two-layer linear and non-linear networks. The advantage of the analysis of variance is that it reveals the effects of initialization, label noise, and training data more clearly than prior approaches. Moreover, we also study the monotonicity and unimodality of the variance components. While prior work studied the unimodality of the overall variance, we study the properties of each term in variance decomposition. One key insight is that in typical settings, the interaction between training samples and initialization can dominate the variance; surprisingly being larger than their marginal effect. Also, we characterize phase transitions where the variance changes from unimodal to monotone. On a technical level, we leverage advanced deterministic equivalent techniques for Haar random matrices, that -- to our knowledge -- have not yet been used in the area. We also verify our results in numerical simulations and on empirical data examples.

研究动机与目标

• 理解过参数化模型中测试误差方差的来源，超越偏差-方差权衡的范畴。
• 利用对称ANOVA框架，将方差分解为标签噪声、初始化和训练数据随机性带来的贡献。
• 分析单个方差分量的单调性与单峰性，而不仅关注总方差。
• 识别泛化误差中训练数据与初始化之间主导的交互效应。
• 表征当模型容量增加时，方差结构从单峰行为向单调行为转变的相变现象。

提出的方法

• 对两层线性与非线性网络应用方差分析（ANOVA），将测试误差方差分解为对称分量。
• 利用哈尓随机矩阵的确定性等价技术，推导方差分量的可处理近似表达式。
• 将测试误差建模为标签噪声、初始化和训练数据实现的函数，以分离其各自贡献。
• 推导方差分解中边际效应与交互效应的解析表达式。
• 通过数值模拟与实证数据示例验证理论发现。
• 追踪不同模型参数化水平下方差分量的演化过程，以检测相变现象。

实验结果

研究问题

• RQ1 在过参数化模型中，标签噪声、初始化和训练数据随机性对测试误差方差的相对贡献是什么？
• RQ2 初始化与训练数据的边际效应与交互效应在量级上如何比较？
• RQ3 在何种条件下，测试误差的方差表现出单峰或单调行为？
• RQ4 随着模型容量的增加，方差结构中发生了哪些相变？
• RQ5 训练数据与初始化之间的交互效应如何影响泛化性能？

主要发现

• 训练数据与初始化之间的交互作用主导了测试误差方差，通常超过其各自边际效应之和。
• 方差分量作为模型参数化程度的函数，表现出单峰行为，且存在明显的相变点，此时方差曲线的形状发生改变。
• 在典型的过参数化设置中，交互效应始终大于初始化或训练数据单独的边际效应。
• ANOVA分解表明，标签噪声对方差有显著贡献，但在大多数配置中并非主导来源。
• 方差行为的相变现象得到了解析表征，并与模型容量和数据分布的变化相关联。
• 数值模拟与实证数据验证了理论预测的方差分解结果及交互作用的主导性。