[论文解读] The Benefits of Over-parameterization at Initialization in Deep ReLU Networks

Devansh Arpit, Yoshua Bengio|arXiv (Cornell University)|Jan 11, 2019

Stochastic Gradient Optimization Techniques参考文献 23被引用 14

一句话总结

本文证明了在初始化时，He初始化的深层ReLU网络表现出范数保持特性：隐藏激活的范数等于输入的范数，且权重梯度的范数等于输入范数与输出误差范数的乘积。这些特性在有限宽度网络中成立，前提是宽度满足推导出的有限下界，从而放宽了以往对无限宽度或i.i.d.输入数据的假设，并通过高概率的PAC分析在有限数据集上得到验证。

ABSTRACT

It has been noted in existing literature that over-parameterization in ReLU networks generally improves performance. While there could be several factors involved behind this, we prove some desirable theoretical properties at initialization which may be enjoyed by ReLU networks. Specifically, it is known that He initialization in deep ReLU networks asymptotically preserves variance of activations in the forward pass and variance of gradients in the backward pass for infinitely wide networks, thus preserving the flow of information in both directions. Our paper goes beyond these results and shows novel properties that hold under He initialization: i) the norm of hidden activation of each layer is equal to the norm of the input, and, ii) the norm of weight gradient of each layer is equal to the product of norm of the input vector and the error at output layer. These results are derived using the PAC analysis framework, and hold true for finitely sized datasets such that the width of the ReLU network only needs to be larger than a certain finite lower bound. As we show, this lower bound depends on the depth of the network and the number of samples, and by the virtue of being a lower bound, over-parameterized ReLU networks are endowed with these desirable properties. For the aforementioned hidden activation norm property under He initialization, we further extend our theory and show that this property holds for a finite width network even when the number of data samples is infinite. Thus we overcome several limitations of existing papers, and show new properties of deep ReLU networks at initialization.

研究动机与目标

• 识别并形式化深层ReLU网络在初始化时的新型理论特性，以改善训练动态。
• 通过推导有限宽度保证，放宽以往研究中常见的无限宽度或i.i.d.输入数据分布的假设。
• 建立激活与梯度范数相等特性在高概率下成立的理论依据，即使在有限数据集上也成立。
• 表明这些有益特性不仅在渐近条件下成立，而且在满足推导出的宽度下界时，对有限宽度网络也成立。

提出的方法

• 使用PAC分析，推导在He初始化下深层ReLU网络中激活与梯度范数的高概率边界。
• 推导出网络宽度的有限下界，使得激活与梯度范数相等特性以高概率成立。
• 应用集中不等式（如Hoeffding型边界）证明隐藏激活的范数在所有层中均接近输入范数。
• 利用联合界和向量投影引理，将范数保持特性扩展至整个网络深度。
• 利用权重矩阵中i.i.d.高斯与伯努利随机变量的性质，建模ReLU网络在初始化时的行为。
• 结合前向传播与反向传播分析，证明梯度范数与输入范数及输出误差范数成比例。

实验结果

研究问题

• RQ1 在有限宽度的深层ReLU网络中，He初始化是否能保持各层之间的激活范数？
• RQ2 能否证明在初始化时，深层ReLU网络的梯度范数与输入范数和输出误差范数成比例？
• RQ3 为使这些范数保持特性以高概率成立，所需的最小有限宽度是多少？
• RQ4 这些特性在不假设无限宽度或i.i.d.输入数据分布的情况下如何成立？

主要发现

• 在每一层，隐藏激活的范数以高概率等于输入的范数，即 \|\mathbf{h}^l\|_2 \approx \|\mathbf{x}\|_2 对所有 l 成立。
• 在每一层，权重梯度的Frobenius范数近似等于输入范数与输出误差范数的乘积，即 \|\partial \ell / \partial \mathbf{W}^l\|_F \approx \|\delta(\mathbf{x},\mathbf{y})\|_2 \cdot \|\mathbf{x}\|_2。
• 存在一个有限的宽度下界，使得这些范数相等特性以高概率成立，其大小取决于网络深度与样本数量。
• 只要宽度超过推导出的有限下界，激活范数相等特性即使在无限数据流中也成立。
• 理论边界在不假设i.i.d.输入数据或无限宽度的前提下推导得出，优于以往的渐近结果。