[论文解读] Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?

Samet Oymak, Mahdi Soltanolkotabi|arXiv (Cornell University)|Dec 25, 2018

Stochastic Gradient Optimization Techniques被引用 56

一句话总结

本论文表明，在过参数化的非线性学习中，梯度下降（以及 SGD）以几何速率收敛到全局最优解，保持在初始化附近，并沿着接近直达初始点的路径，趋向于接近初始化的全局最优解。

ABSTRACT

Many modern learning tasks involve fitting nonlinear models to data which are trained in an overparameterized regime where the parameters of the model exceed the size of the training dataset. Due to this overparameterization, the training loss may have infinitely many global minima and it is critical to understand the properties of the solutions found by first-order optimization schemes such as (stochastic) gradient descent starting from different initializations. In this paper we demonstrate that when the loss has certain properties over a minimally small neighborhood of the initial point, first order methods such as (stochastic) gradient descent have a few intriguing properties: (1) the iterates converge at a geometric rate to a global optima even when the loss is nonconvex, (2) among all global optima of the loss the iterates converge to one with a near minimal distance to the initial point, (3) the iterates take a near direct route from the initial point to this global optima. As part of our proof technique, we introduce a new potential function which captures the precise tradeoff between the loss function and the distance to the initial point as the iterations progress. For Stochastic Gradient Descent (SGD), we develop novel martingale techniques that guarantee SGD never leaves a small neighborhood of the initialization, even with rather large learning rates. We demonstrate the utility of our general theory for a variety of problem domains spanning low-rank matrix recovery to neural network training. Underlying our analysis are novel insights that may have implications for training and generalization of more sophisticated learning problems including those involving deep neural network architectures.

研究动机与目标

• 在过参数化的非线性学习设置中激发研究动机并分析训练动力学。
• 在较温和的局部雅可比矩阵假设下刻画梯度下降与 SGD 的收敛行为。
• 证明梯度方法对数据进行插值并收敛到全局最优、接近初始化的解。
• 证明可应用于广义线性模型、低秩回归和浅层神经网络。

提出的方法

• 建立非线性最小二乘问题，并通过雅可比矩阵表达梯度。
• 在局部邻域内对雅可比矩阵谱及雅可比矩阵偏差施加假设。
• 在上述假设下证明梯度下降对全局最优解的线性收敛。
• 利用鞅方法证明 SGD 在保持在初始化邻域内的同时以高概率收敛。
• 将该通用理论应用于广义线性模型、低秩回归和浅层神经网络。

实验结果

研究问题

• RQ1 在过参数化的非线性学习中，梯度下降和 SGD 在何种条件下收敛到全局最优解？
• RQ2 梯度方法是否选择接近初始化的全局最优解，并且是否沿着从初始化到最优解的短而直接的路径？
• RQ3 雅可比谱及其局部偏差如何影响收敛和轨迹？
• RQ4 该理论能否在广义线性模型、低秩回归和浅层神经网络中得到实例化？
• RQ5 在过参数化范畴中，对插值、泛化和训练动力学有何意义？

主要发现

• 在局部雅可比假设下，梯度下降在非凸的过参数化设置下以几何收敛到全局最优解。
• 在所有全局最优解中，梯度下降收敛到最接近初始化的那一个。
• 梯度路径的总长度有上界，意味着从初始化到全局最优解的轨迹接近直线。
• SGD 以线性收敛收敛并且在很大学习率下仍以高概率保持在初始化的一个小邻域内。
• 该理论在广义线性模型、低秩矩阵回归和浅层神经网络训练中得到证明。