[논문 리뷰] Escaping Saddles with Stochastic Gradients

Hadi Daneshmand, Jonas Köhler외 2명|arXiv (Cornell University)|2018. 03. 15.

Stochastic Gradient Optimization Techniques참고 문헌 23인용 수 57

한 줄 요약

논문은 확률적 경사가 음의 곡률을 상관관계로 가지며 등방성 노이즈 없이도 saddle을 벗어날 수 있음을 보이고 CNC 가정하에 SGD의 차원 독립적인 2차 수렴 속도를 제시한다.

ABSTRACT

We analyze the variance of stochastic gradients along negative curvature directions in certain non-convex machine learning models and show that stochastic gradients exhibit a strong component along these directions. Furthermore, we show that - contrary to the case of isotropic noise - this variance is proportional to the magnitude of the corresponding eigenvalues and not decreasing in the dimensionality. Based upon this observation we propose a new assumption under which we show that the injection of explicit, isotropic noise usually applied to make gradient descent escape saddle points can successfully be replaced by a simple SGD step. Additionally - and under the same condition - we derive the first convergence rate for plain SGD to a second-order stationary point in a number of iterations that is independent of the problem dimension.

연구 동기 및 목표

• Motivate the challenge of escaping saddle points in non-convex optimization with SGD.
• Introduce the Correlated Negative Curvature (CNC) assumption for stochastic gradients.
• Show that SGD can converge to a second-order stationary point without isotropic perturbations.
• Provide convergence rates that are independent of problem dimensionality under CNC.
• Validate CNC theoretically for learning half-spaces and empirically on neural networks.

제안 방법

• Define CNC: the stochastic gradient projection on the minimum Hessian eigenvector has uniformly bounded away from zero second moment (gamma).
• Study GD perturbed by SGD steps (CNC-PGD) and SGD without perturbation (CNC-SGD) under smoothness assumptions.
• Prove Theorem 1: CNC-PGD finds an (epsilon, sqrt(rho) epsilon^{2/5})-second-order stationary point in O((ell L)^4 (delta gamma epsilon)^{-2} log(...)) steps with high probability.
• Prove Theorem 2: CNC-SGD finds an (epsilon, sqrt(rho) epsilon)-second-order stationary point in O((L^3 ell^{10})/(delta^4 gamma^4) * epsilon^{-10} log^2(...)) steps with high probability.
• Show CNC holds for learning half-spaces (via a lower bound on projected gradient variance).
• Provide empirical evidence that stochastic gradients have significant variance along negative curvature directions in neural networks.]
• research_questions:[

실험 결과

연구 질문

• RQ1 Can SGD escape saddle points without explicit isotropic noise under a weaker CNC assumption?
• RQ2 What convergence rates to second-order stationary points are attainable by CNC-PGD and CNC-SGD, and are these rates dimension-dependent?
• RQ3 Does the CNC condition hold for practical problems like learning half-spaces and training neural networks?
• RQ4 How does the variance of stochastic gradients along negative curvature directions behave with respect to Hessian eigenvalues and network width/depth?
• RQ5 Do empirical results on neural networks support the CNC hypothesis and its implications for optimization dynamics?

주요 결과

• Under CNC, CNC-PGD achieves an (epsilon, sqrt(rho) epsilon^{2/5})-second-order stationary point in poly(log) iterations and without explicit isotropic noise.
• Under CNC, CNC-SGD reaches an (epsilon, sqrt(rho) epsilon)-second-order stationary point in roughly epsilon^{-10} iterations, with dimension-free convergence.
• Stochastic gradients exhibit a strong component along negative curvature directions, with variance along these directions proportional to the corresponding eigenvalues, not diminishing with dimension.
• For learning half-spaces, CNC holds for the stochastic gradient, enabling the convergence guarantees without added perturbations.
• Empirical results on MNIST indicate stochastic gradients retain variance along minimum-curvature directions independent of network width/depth, supporting CNC applicability.