[論文レビュー] Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?

Samet Oymak, Mahdi Soltanolkotabi|arXiv (Cornell University)|Dec 25, 2018

Stochastic Gradient Optimization Techniques被引用数 56

ひとこと要約

この論文は、過parameterized nonlinear learning において、gradient descent (and SGD) が幾何的収束率でグローバル最適解へ収束し、初期化の近傍にとどまり、初期化に近い globally optimal な解へほぼ直接的な経路をたどることを示している。

ABSTRACT

Many modern learning tasks involve fitting nonlinear models to data which are trained in an overparameterized regime where the parameters of the model exceed the size of the training dataset. Due to this overparameterization, the training loss may have infinitely many global minima and it is critical to understand the properties of the solutions found by first-order optimization schemes such as (stochastic) gradient descent starting from different initializations. In this paper we demonstrate that when the loss has certain properties over a minimally small neighborhood of the initial point, first order methods such as (stochastic) gradient descent have a few intriguing properties: (1) the iterates converge at a geometric rate to a global optima even when the loss is nonconvex, (2) among all global optima of the loss the iterates converge to one with a near minimal distance to the initial point, (3) the iterates take a near direct route from the initial point to this global optima. As part of our proof technique, we introduce a new potential function which captures the precise tradeoff between the loss function and the distance to the initial point as the iterations progress. For Stochastic Gradient Descent (SGD), we develop novel martingale techniques that guarantee SGD never leaves a small neighborhood of the initialization, even with rather large learning rates. We demonstrate the utility of our general theory for a variety of problem domains spanning low-rank matrix recovery to neural network training. Underlying our analysis are novel insights that may have implications for training and generalization of more sophisticated learning problems including those involving deep neural network architectures.

研究の動機と目的

• Motivate and analyze training dynamics in overparameterized nonlinear learning settings.
• Characterize convergence behavior of gradient descent and SGD under mild local Jacobian assumptions.
• Show that gradient methods interpolate data and converge to the globally optimal, initialization-near solution.
• Demonstrate applicability to generalized linear models, low-rank regression, and shallow neural networks.

提案手法

• Formulate nonlinear least-squares problems and express gradient via the Jacobian.
• Impose Assumptions on Jacobian spectrum and Jacobian deviations in a local neighborhood.
• Prove linear convergence of gradient descent to a global optimum under the assumptions.
• Prove SGD converges with high probability while remaining in a neighborhood of initialization using martingale techniques.
• Apply the general theory to generalized linear models, low-rank regression, and shallow neural networks.

実験結果

リサーチクエスチョン

• RQ1 Under what conditions do gradient descent and SGD converge to a global optimum in overparameterized nonlinear learning?
• RQ2 Do gradient methods select global optima close to initialization, and do they trace a short, direct path from initialization to the optimum?
• RQ3 How does the Jacobian spectrum and its local deviations influence convergence and trajectory?
• RQ4 Can the theory be instantiated for GLMs, low-rank regression, and shallow neural networks?
• RQ5 What are the implications for interpolation, generalization, and training dynamics in overparameterized regimes?

主な発見

• Gradient descent converges geometrically to a global optimum in nonconvex overparameterized settings under local Jacobian assumptions.
• Among all global optima, gradient descent converges to one closest to the initialization.
• The total gradient path length is bounded, implying a near-direct trajectory from initialization to the global optimum.
• SGD converges linearly and remains in a small neighborhood of initialization with high probability, even with relatively large learning rates.
• The theory is demonstrated across generalized linear models, low-rank matrix regression, and shallow neural network training.