[Paper Review] Convergence Rates for the Stochastic Gradient Descent Method for Non-Convex Objective Functions

Benjamin Fehrman, Benjamin Gess+1 more|arXiv (Cornell University)|Apr 2, 2019

Stochastic Gradient Optimization Techniques51 references19 citations

TL;DR

This paper establishes convergence rates for stochastic gradient descent (SGD) on non-convex objective functions without requiring global convexity or contraction assumptions. It proves local convergence to minima with explicit probabilistic bounds, applicable to neural networks with symmetries or non-unique minima, under local smoothness and nondegeneracy conditions on the Hessian at minima.

ABSTRACT

We prove the local convergence to minima and estimates on the rate of convergence for the stochastic gradient descent method in the case of not necessarily globally convex nor contracting objective functions. In particular, the results are applicable to simple objective functions arising in machine learning.

Motivation & Objective

• Address the lack of convergence rate guarantees for SGD in non-convex machine learning problems, where global convexity or contraction assumptions do not hold.
• Analyze SGD convergence locally at sets of minima, rather than globally, to accommodate non-unique or manifold-shaped minima common in neural networks.
• Establish quantitative convergence rates for SGD under local smoothness and nondegeneracy conditions on the Hessian at minima.
• Provide probabilistic bounds on the likelihood that the algorithm output is an ε-suboptimal solution, even when minima are not isolated.
• Demonstrate applicability to simple neural networks with ReLU or linear activations, where minima form manifolds due to weight symmetries.

Proposed method

• Formulate the objective function as an expectation: f(θ) = E[F(θ, X)], where F is measurable and X is a random variable.
• Assume local C³ smoothness of f in a neighborhood of the minimum set M ∩ U, and that M ∩ U is a C¹ submanifold of dimension d.
• Impose a nondegeneracy condition: the Hessian of f has rank d − d at every point in M ∩ U, ensuring the minimum set is not degenerate.
• Use a mini-batch SGD scheme with independent trajectories: Θk,M,r_n = Θk,M,r_{n−1} − (r / (n^{ρ} M)) Σ_{m=1}^M ∇θF(Θk,M,r_{n−1}, Xk,n,m), for ρ ∈ (2/3, 1).
• Select the best-performing trajectory among K independent runs at each step using a mini-batch loss evaluation: ΘK,M,M,r_n minimizes Σ_{m=1}^M F(Θk,M,r_n, X1,n+1,m).
• Derive a high-probability bound on the suboptimality: P(f(ΘK,M,M,r_n) − inf f ≥ ε) ≤ cK/ε²M + [κ + c(1/(ε²n^ρ) + n^{1−ρ}/M^{1/2})]^K, with κ ∈ [0,1).

Experimental results

Research questions

• RQ1 Can convergence rates for SGD be established in the absence of global convexity or contraction assumptions on the objective function?
• RQ2 What conditions ensure local convergence of SGD to a set of minima when minima are not isolated points but form a manifold?
• RQ3 How can convergence be quantified when the Hessian at minima is degenerate in the directions transverse to the minimum manifold?
• RQ4 What is the trade-off between mini-batch size M, number of independent runs K, and iteration count n in achieving ε-suboptimal solutions with high probability?
• RQ5 Can the theoretical framework be applied to simple neural networks with ReLU or linear activations, where minima are non-unique due to symmetry?

Key findings

• The paper establishes a high-probability bound on the suboptimality of the best-of-K SGD trajectories: P(f(ΘK,M,M,r_n) − inf f ≥ ε) ≤ cK/ε²M + [κ + c(1/(ε²n^ρ) + n^{1−ρ}/M^{1/2})]^K.
• The bound decays exponentially in K, with a rate controlled by κ ∈ [0,1), which captures the probability of initial conditions lying outside basins of attraction and failure within basins.
• The term cK/ε²M vanishes as M → ∞, corresponding to exact gradient computation, showing that mini-batch size M controls the bias from stochastic approximation.
• The convergence rate depends on n^ρ and n^{1−ρ}, with ρ ∈ (2/3, 1), indicating a trade-off between step size decay and variance reduction.
• For the four-parameter linear network, the minimum set M ∩ U is a 2-dimensional C¹ submanifold, and the Hessian has full rank 2 = codim(M ∩ U), satisfying the nondegeneracy condition.
• For the two-parameter ReLU network, the minimum set M ∩ U is a 1-dimensional C¹ submanifold, and the Hessian has rank 1 = codim(M ∩ U), confirming the nondegeneracy condition holds.