[论文解读] Effect of Activation Functions on the Training of Overparametrized Neural Nets

Abhishek Panigrahi, Abhishek Shetty+1 more|arXiv (Cornell University)|Aug 16, 2019

Stochastic Gradient Optimization Techniques参考文献 50被引用 12

一句话总结

本文研究了不同激活函数（ReLU、ELU、Swish 和 tanh）对合成数据上过参数化两层神经网络训练动态的影响。在使用均方误差损失和固定输出权重的条件下，发现 ReLU 最快收敛至零训练误差，其次是 ELU、Swish 和 tanh，且收敛速度与初始化时 Gram 矩阵的特征值密切相关。

ABSTRACT

It is well-known that overparametrized neural networks trained using gradient-based methods quickly achieve small training error with appropriate hyperparameter settings. Recent papers have proved this statement theoretically for highly overparametrized networks under reasonable assumptions. These results either assume that the activation function is ReLU or they crucially depend on the minimum eigenvalue of a certain Gram matrix depending on the data, random initialization and the activation function. In the later case, existing works only prove that this minimum eigenvalue is non-zero and do not provide quantitative bounds. On the empirical side, a contemporary line of investigations has proposed a number of alternative activation functions which tend to perform better than ReLU at least in some settings but no clear understanding has emerged. This state of affairs underscores the importance of theoretically understanding the impact of activation functions on training. In the present paper, we provide theoretical results about the effect of activation function on the training of highly overparametrized 2-layer neural networks. A crucial property that governs the performance of an activation is whether or not it is smooth. For non-smooth activations such as ReLU, SELU and ELU, all eigenvalues of the associated Gram matrix are large under minimal assumptions on the data. For smooth activations such as tanh, swish and polynomials, the situation is more complex. If the subspace spanned by the data has small dimension then the minimum eigenvalue of the Gram matrix can be small leading to slow training. But if the dimension is large and the data satisfies another mild condition, then the eigenvalues are large. If we allow deep networks, then the small data dimension is not a limitation provided that the depth is sufficient. We discuss a number of extensions and applications of these results.

研究动机与目标

• 理解激活函数选择对过参数化神经网络训练动态的影响。
• 研究激活函数是否会影响过参数化设置下的收敛速度。
• 分析初始化时 Gram 矩阵特征值在决定训练速度中的作用。
• 在合成数据上通过多轮实验对比 ReLU、ELU、Swish 和 tanh 的表现，标签设置受控。
• 验证理论预期：即初始特征值大小与过参数化设置下收敛行为之间的关系。

提出的方法

• 使用第一层包含 10^6 个神经元的两层神经网络，采用均方误差损失进行训练。
• 训练过程中固定输出层权重，以隔离第一层激活的影响。
• 使用合成数据：在 S^1 上等距分布的 n 个点，映射到 S^9 并随机标记为 ±1。
• 通过在多个训练周期内平均 5 次随机运行的平方损失来评估收敛性。
• 分析初始化时的 Gram 矩阵 G，研究不同激活函数下特征值分布。
• 比较特征值大小及其在训练过程中的演化，以解释收敛速度的差异。

实验结果

研究问题

• RQ1 激活函数的选择如何影响过参数化两层网络中的收敛速度？
• RQ2 初始化时 Gram 矩阵中较大的特征值是否与更快的收敛速度相关？
• RQ3 ReLU、ELU、Swish 和 tanh 在训练损失随时间减少方面有何差异？
• RQ4 过参数化设置是否表现出训练过程中特征值和特征向量的稳定性？
• RQ5 初始 Gram 矩阵的特性是否能预测不同激活函数之间的相对训练性能？

主要发现

• ReLU 显著快于 ELU、Swish 和 tanh 收敛至零训练误差。
• ELU 的收敛速度优于 Swish 和 tanh，但慢于 ReLU。
• 初始化时，ReLU 和 ELU 的 Gram 矩阵特征值最大，而 tanh 和 Swish 的最小。
• 特征值和特征向量在整个训练过程中几乎保持不变，证实了过参数化状态。
• 收敛速度排序（ReLU > ELU > Swish > tanh）与初始 Gram 矩阵特征值的大小直接相关。
• 观察到的训练动态与理论预期一致，即初始 Gram 矩阵结构决定了优化速度。