[论文解读] Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex

Hongyang Zhang, Junru Shao+1 more|arXiv (Cornell University)|Jun 6, 2018

Adversarial Robustness in Machine Learning参考文献 56被引用 9

一句话总结

该论文表明，具有多分支结构的深度神经网络表现出更低的非凸性，这通过对偶间隙的缩小来量化。论文证明，对于任意激活函数和铰链损失，随着分支数量的增加，总体风险和经验风险的对偶间隙均趋近于零；而对于具有ℓ₂损失的线性网络，对偶间隙恰好为零，意味着强对偶性成立。

ABSTRACT

Several recently proposed architectures of neural networks such as ResNeXt, Inception, Xception, SqueezeNet and Wide ResNet are based on the designing idea of having multiple branches and have demonstrated improved performance in many applications. We show that one cause for such success is due to the fact that the multi-branch architecture is less non-convex in terms of duality gap. The duality gap measures the degree of intrinsic non-convexity of an optimization problem: smaller gap in relative value implies lower degree of intrinsic non-convexity. The challenge is to quantitatively measure the duality gap of highly non-convex problems such as deep neural networks. In this work, we provide strong guarantees of this quantity for two classes of network architectures. For the neural networks with arbitrary activation functions, multi-branch architecture and a variant of hinge loss, we show that the duality gap of both population and empirical risks shrinks to zero as the number of branches increases. This result sheds light on better understanding the power of over-parametrization where increasing the network width tends to make the loss surface less non-convex. For the neural networks with linear activation function and $\ell_2$ loss, we show that the duality gap of empirical risk is zero. Our two results work for arbitrary depths and adversarial data, while the analytical techniques might be of independent interest to non-convex optimization more broadly. Experiments on both synthetic and real-world datasets validate our results.

研究动机与目标

• 理解为何ResNeXt、Inception和Wide ResNet等多分支神经网络架构能实现更优性能。
• 利用对偶间隙作为度量，量化深度神经网络的内在非凸性。
• 为多分支网络和线性神经网络中的总体风险和经验风险，建立对偶间隙的理论保证。
• 提供关于过参数化和网络架构设计在降低优化难度方面优势的解析性洞见。

提出的方法

• 利用Shapley-Folkman引理，对具有任意激活函数和铰链损失的多分支网络中的对偶间隙进行有界。
• 证明随着分支数量的增加，总体风险和经验风险的对偶间隙趋于零。
• 通过构造对偶证书并应用低秩逼近，建立具有ℓ₂损失的深层线性网络中的强对偶性。
• 推导出非凸优化公式的对偶问题，表明其为凸问题，可通过近端算法求解。
• 使用截断奇异值分解（SVD）和投影算子分析最优解的结构。
• 运用矩阵范数不等式和谱性质，对原始解与对偶解之间的残差进行有界。

实验结果

研究问题

• RQ1 深度神经网络中的多分支结构是否会导致内在非凸性的降低？
• RQ2 能否在具有任意激活函数的深度神经网络中，对对偶间隙进行解析有界或最小化？
• RQ3 在具有ℓ₂损失的深层线性网络中，强对偶性（对偶间隙为零）在何种条件下成立？
• RQ4 增加分支数量如何影响总体风险和经验风险设置下的对偶间隙？
• RQ5 过参数化在降低深度学习优化景观的非凸性方面起到什么作用？

主要发现

• 对于具有任意激活函数和铰链损失变体的深度神经网络，随着分支数量的增加，总体风险和经验风险的对偶间隙均缩小至零。
• 对于具有ℓ₂损失的深层线性神经网络，经验风险的对偶间隙恰好为零，意味着强对偶性成立。
• 理论结果对任意网络深度和对抗性数据分布均成立。
• 对偶间隙的降低归因于多分支结构，该结构使优化景观的非凸性降低。
• 在合成数据集和真实世界数据集上的实验验证了理论发现，表明损失曲面更加平滑且非凸性降低。
• 包括对偶证书构造和低秩逼近在内的分析技术，对非凸优化研究本身也具有独立兴趣。