[论文解读] On Warm-Starting Neural Network Training

Jordan T. Ash, Ryan P. Adams|arXiv (Cornell University)|Oct 18, 2019

Machine Learning and Data Classification参考文献 51被引用 56

一句话总结

本文表明在序列数据设定中对神经网络进行热启动会损害泛化能力，并提出一个简单的先收缩再扰动的技巧，在保持更快训练速度的同时缩小泛化差距。

ABSTRACT

In many real-world deployments of machine learning systems, data arrive piecemeal. These learning scenarios may be passive, where data arrive incrementally due to structural properties of the problem (e.g., daily financial data) or active, where samples are selected according to a measure of their quality (e.g., experimental design). In both of these cases, we are building a sequence of models that incorporate an increasing amount of data. We would like each of these models in the sequence to be performant and take advantage of all the data that are available to that point. Conventional intuition suggests that when solving a sequence of related optimization problems of this form, it should be possible to initialize using the solution of the previous iterate -- to "warm start" the optimization rather than initialize from scratch -- and see reductions in wall-clock time. However, in practice this warm-starting seems to yield poorer generalization performance than models that have fresh random initializations, even though the final training losses are similar. While it appears that some hyperparameter settings allow a practitioner to close this generalization gap, they seem to only do so in regimes that damage the wall-clock gains of the warm start. Nevertheless, it is highly desirable to be able to warm-start neural network training, as it would dramatically reduce the resource usage associated with the construction of performant deep learning systems. In this work, we take a closer look at this empirical phenomenon and try to understand when and how it occurs. We also provide a surprisingly simple trick that overcomes this pathology in several important situations, and present experiments that elucidate some of its properties.

研究动机与目标

• 激发并分析在序列数据更新情境下对神经网络进行热启动时观察到的泛化差距。
• 量化泛化差距在不同架构、优化器和数据集上的表现。
• 在热启动情境下评估常规初始化/正则化方法。
• 提出并验证一种简单的纠正技术以缓解热启动引发的泛化差距。
• 探讨在数据丰富的情景中在线学习和预训练的意义。

提出的方法

• 在不同架构（ResNet-18、MLP、逻辑回归）和数据集（CIFAR-10、CIFAR-100、SVHN）之间，对热启动网络与随机初始化网络进行经验比较。
• 在数据分块到来并使用上一次参数更新模型与从新开始更新的在线/批量更新实验。
• 对批量大小和学习率进行系统的超参数扫描，以评估潜在的纠正措施。
• 进行正则化实验以测试L2、置信惩罚或对抗性训练是否能缓解差距。
• 引入并评估 shrink, perturb, repeat 技巧，使用受控的收缩因子 lambda 和噪声 sigma 以平衡梯度。
• 对梯度大小的分析，以理解该技巧背后的机制。

实验结果

研究问题

• RQ1 在使用新数据更新神经网络时，热启动是否相较于随机初始化降低泛化能力？
• RQ2 泛化差距如何随模型类型、数据集难度和优化器而变化？
• RQ3 常规方法（正则化、超参数调整）能否在热启动训练中解决该差距？
• RQ4 一种简单的初始化修改（收缩-扰动）能否在序列更新中恢复泛化并保持更快的训练？
• RQ5 在数据丰富的情景中，预训练迁移与热启动之间的相互作用是什么？

主要发现

• 热启动的神经网络的一致表现为泛化能力劣于随机初始化的同类，即使训练准确度相似。
• 泛化差距在 CIFAR-10 上比 SVHN 更明显；逻辑回归（凸损失）受到的影响较小。
• 更改批量大小或学习率都不能完全解决差距；在某些情况下，热启动模型会遗忘之前的学习，但训练速度更快。
• 一个简单的收缩-扰动技巧 (theta^t = lambda theta^{t-1} + p^t, p^t ~ N(0, sigma^2)) 能缩小差距并加速在线训练，lambda 约为 0.6 时通常与随机初始化的表现相匹配。
• 收缩在一定程度上保留了学习到的假设，平衡了旧数据与新数据之间的梯度贡献，并且在与扰动结合时，相较于纯热启动，改善了训练效率和泛化能力。
• 来自不相关数据的预训练迁移在数据丰富的第二任务中可能降低性能；收缩-扰动初始化在迁移情景中具有自适应性，视数据可用性而至少与热启动或从零训练表现相当。