[论文解读] Stationary Points of Shallow Neural Networks with Quadratic Activation Function

David Gamarnik, Eren C. Kızıldağ+1 more|arXiv (Cornell University)|Dec 3, 2019

Stochastic Gradient Optimization Techniques参考文献 62被引用 8

一句话总结

本文在教师-学生框架下分析了具有二次激活函数的浅层神经网络的优化景观。研究证明，当初始化低于可量化的能量屏障时，梯度下降法可在多项式时间内收敛至全局最优解；并表明在过参数化条件下，通过缩放的单位矩阵进行随机初始化可确保满足该初始化条件，该结论基于随机矩阵理论。

ABSTRACT

We consider the teacher-student setting of learning shallow neural networks with quadratic activations and planted weight matrix $W^*\in\mathbb{R}^{m imes d}$, where $m$ is the width of the hidden layer and $d\le m$ is the data dimension. We study the optimization landscape associated with the empirical and the population squared risk of the problem. Under the assumption the planted weights are full-rank we obtain the following results. First, we establish that the landscape of the empirical risk admits an "energy barrier" separating rank-deficient $W$ from $W^*$: if $W$ is rank deficient, then its risk is bounded away from zero by an amount we quantify. We then couple this result by showing that, assuming number $N$ of samples grows at least like a polynomial function of $d$, all full-rank approximate stationary points of the empirical risk are nearly global optimum. These two results allow us to prove that gradient descent, when initialized below the energy barrier, approximately minimizes the empirical risk and recovers the planted weights in polynomial-time. Next, we show that initializing below this barrier is in fact easily achieved when the weights are randomly generated under relatively weak assumptions. We show that provided the network is sufficiently overparametrized, initializing with an appropriate multiple of the identity suffices to obtain a risk below the energy barrier. At a technical level, the last result is a consequence of the semicircle law for the Wishart ensemble and could be of independent interest. Finally, we study the minimizers of the empirical risk and identify a simple necessary and sufficient geometric condition on the training data under which any minimizer has necessarily zero generalization error. We show that as soon as $N\ge N^*=d(d+1)/2$, randomly generated data enjoys this geometric condition almost surely, while that ceases to be true if $N

研究动机与目标

• 理解教师-学生设置下具有二次激活函数的浅层神经网络的优化景观。
• 确定梯度下降收敛至全局最优解的条件。
• 刻画初始化在避免不良局部极小值中的作用。
• 确定实现零泛化误差的训练样本临界数量。
• 建立使最小化器实现零泛化误差的数据几何条件。

提出的方法

• 在植入学得权重矩阵为满秩的假设下，分析经验风险与总体风险函数。
• 引入一个能量屏障，将秩亏的权重矩阵与真实权重矩阵 $W^*$ 分离，量化秩亏 $W$ 的经验风险下限。
• 利用随机矩阵理论，特别是Wishart系综的半圆律，证明以缩放单位矩阵初始化时，风险低于能量屏障的概率很高。
• 证明当样本数 $N$ 关于 $d$ 多项式增长时，经验风险的所有满秩近似驻点均为近似全局最优解。
• 基于 $X_i X_i^T$ 的张成空间，推导出使最小化器实现零泛化误差的必要且充分几何条件。
• 运用线性代数与随机矩阵理论工具，包括迹不等式与谱范数界，推导风险下限。

实验结果

研究问题

• RQ1 在何种条件下，梯度下降可在具有二次激活函数的浅层网络中收敛至全局最优解？
• RQ2 在非凸优化景观中，初始化在避免不良局部极小值中起到何种作用？
• RQ3 实现零泛化误差的训练样本临界数量 $N^*$ 是多少？
• RQ4 训练数据的何种几何条件可确保所有最小化器均实现零泛化误差？
• RQ5 对于秩亏权重矩阵，经验风险的行为如何？是否可被量化？

主要发现

• 对于秩亏 $W$，经验风险 $\widehat{\mathcal{L}}(W)$ 被一个可量化的能量屏障与真实权重矩阵 $W^*$ 明确分离，且其值远离零。
• 当 $N$ 关于 $d$ 至少多项式增长时，$\widehat{\mathcal{L}}(W)$ 的所有满秩近似驻点均为近似全局最优解。
• 在能量屏障以下初始化的梯度下降可近似最小化经验风险，并在多项式时间内恢复 $W^*$。
• 在充分过参数化条件下，以适当倍数的单位矩阵进行随机初始化，可使风险低于能量屏障，且该结果以高概率成立。
• 实现零泛化误差的临界样本数为 $N^* = d(d+1)/2$，当 $N \geq N^*$ 时，该阈值在随机数据下几乎必然满足。
• 零泛化误差的必要且充分几何条件是 $\{X_i X_i^T\}$ 的张成空间等于所有对称 $d \times d$ 矩阵的空间，当 $N \geq d(d+1)/2$ 时，该条件几乎必然成立。