[论文解读] Skip Connections Eliminate Singularities

A. Emin Orhan, Xaq Pitkow|arXiv (Cornell University)|Jan 31, 2017

Stochastic Gradient Optimization Techniques被引用 86

一句话总结

本文认为跳跃连接通过消除损失面中的不可辨识性奇点（消除、重叠和线性依赖）来提升深度网络的训练，理论分析与跨深度网络和数据集的实证结果提供支持。

ABSTRACT

Skip connections made the training of very deep networks possible and have become an indispensable component in a variety of neural architectures. A completely satisfactory explanation for their success remains elusive. Here, we present a novel explanation for the benefits of skip connections in training very deep networks. The difficulty of training deep networks is partly due to the singularities caused by the non-identifiability of the model. Several such singularities have been identified in previous works: (i) overlap singularities caused by the permutation symmetry of nodes in a given layer, (ii) elimination singularities corresponding to the elimination, i.e. consistent deactivation, of nodes, (iii) singularities generated by the linear dependence of the nodes. These singularities cause degenerate manifolds in the loss landscape that slow down learning. We argue that skip connections eliminate these singularities by breaking the permutation symmetry of nodes, by reducing the possibility of node elimination and by making the nodes less linearly dependent. Moreover, for typical initializations, skip connections move the network away from the "ghosts" of these singularities and sculpt the landscape around them to alleviate the learning slow-down. These hypotheses are supported by evidence from simplified models, as well as from experiments with deep networks trained on real-world datasets.

研究动机与目标

• 阐述并解释为何训练非常深的网络受益于跳跃连接。
• 识别并表征三种会减慢学习的奇点（消除、重叠、线性依赖）。
• 证明跳跃连接在不同体系结构和数据集上减少简并性并加速训练。
• 提供超越标准残差的实用替代方案和架构洞见，以进一步缓解奇点。

提出的方法

• 对全连接层中的三种奇点进行模型分析：消除、重叠和线性依赖。
• 理论讨论跳跃连接如何破坏奇异流形并恢复可辨识性。
• 在 CIFAR-100/CIFAR-10/100 上对普通网络、残差网络和超残差网络进行经验比较，并进行 Hessian 特征值密度估计。
• 引入 BiasReg，作为一种简单的偏置目标正则化，用以打破置换对称性并消除奇点。
• 评估非单位跳跃方案，包括随机致密正交跳跃，以测试对称性破坏效应。
• 研究梯度范数和梯度消失现象，包含批量归一化对 BiasReg 网络的影响。

实验结果

研究问题

• RQ1 跳跃连接是否能够消除深度网络中的不可辨识性奇点？
• RQ2 消除、重叠和线性依赖三种奇点如何影响学习动力学和优化景观？
• RQ3 跳跃连接是否在有利的初始化之外提升训练速度和鲁棒性？
• RQ4 替代的对称性破坏方法（如 BiasReg、正交跳跃）能否复制跳跃连接的效益？

主要发现

• 跳跃连接降低了 Hessian 谱的简并性，与相较于纯网络更快的训练相关。
• 超残差体系结构在研究的架构中表现出最低的简并性和最高的早期训练速度。
• 针对偏置的偏置正则化以打破对称性提升了相对于纯网络的性能，尽管尚未完全达到残差的水平。
• 正交（密集）跳跃连接在更好地区分单元、降低消除/重叠风险方面略优于单位跳跃。
• 恶意初始化结果表明跳跃连接的好处不仅限于初始化，指向奇点附近地形的重塑。
• 来自浅层和深层网络的证据表明奇点是优化中的一个有意义的瓶颈，跳跃连接可以缓解。