[Paper Review] Distributed Stochastic Gradient Descent: Nonconvexity, Nonsmoothness, and Convergence to Local Minima

Brian R. Swenson, Ryan Murray+2 more|arXiv (Cornell University)|Mar 5, 2020

Stochastic Gradient Optimization Techniques16 citations

TL;DR

This paper establishes theoretical convergence guarantees for distributed stochastic gradient descent (D-SGD) in nonconvex, nonsmooth optimization settings. By modeling D-SGD via a continuous-time distributed gradient flow and extending stable manifold theory to nonautonomous systems, it proves that D-SGD converges to critical points almost surely and avoids saddle points under mild smoothness conditions near saddle points, mirroring centralized SGD behavior.

ABSTRACT

In centralized settings, it is well known that stochastic gradient descent (SGD) avoids saddle points and converges to local minima in nonconvex problems. However, similar guarantees are lacking for distributed first-order algorithms. The paper studies distributed stochastic gradient descent (D-SGD)--a simple network-based implementation of SGD. Conditions under which D-SGD avoids saddle points and converges to local minima are studied. First, we consider the problem of computing critical points. Assuming loss functions are nonconvex and possibly nonsmooth, it is shown that, for each fixed initialization, D-SGD converges to critical points of the loss with probability one. Next, we consider the problem of avoiding saddle points. In this case, we again assume that loss functions may be nonconvex and nonsmooth, but are smooth in a neighborhood of a saddle point. It is shown that, for any fixed initialization, D-SGD avoids such saddle points with probability one. Results are proved by studying the underlying (distributed) gradient flow, using the ordinary differential equation (ODE) method of stochastic approximation, and extending classical techniques from dynamical systems theory such as stable manifolds. Results are proved in the general context of subspace-constrained optimization, of which D-SGD is a special case.

Motivation & Objective

• To close the theoretical gap in distributed first-order optimization by establishing convergence to critical points and avoidance of saddle points for D-SGD.
• To extend classical stochastic approximation and dynamical systems techniques to nonautonomous, nonclassical systems arising in distributed optimization.
• To analyze D-SGD under general nonconvex and possibly nonsmooth loss functions, including those arising in neural networks with nonsmooth activations.
• To provide rigorous conditions under which D-SGD avoids saddle points, even when the global loss is nonsmooth, provided smoothness holds in a neighborhood of the saddle point.
• To generalize existing convergence results from centralized to distributed settings, particularly for subspace-constrained optimization.

Proposed method

• Model D-SGD as a continuous-time distributed gradient flow (DGF) using the ordinary differential equation (ODE) method of stochastic approximation.
• Apply stable manifold theory to nonautonomous dynamical systems to analyze nonconvergence to saddle points.
• Use a linearization approach around saddle points and study eigenvalue convergence of the Hessian under time-varying dynamics.
• Introduce a quadratic penalty formulation to handle subspace constraints in the optimization problem.
• Establish convergence to critical points under local Lipschitz continuity and nonconvexity of individual agent functions.
• Prove almost sure convergence to critical points and almost sure avoidance of regular saddle points using measure-theoretic arguments on stable manifolds.

Experimental results

Research questions

• RQ1 Under what conditions does distributed stochastic gradient descent converge to critical points when the loss functions are nonconvex and possibly nonsmooth?
• RQ2 Can D-SGD avoid saddle points in nonconvex optimization, even when the global loss is nonsmooth, provided smoothness holds near the saddle point?
• RQ3 How can classical stable manifold theory be extended to nonautonomous systems arising in distributed gradient flows?
• RQ4 What are the necessary and sufficient conditions for D-SGD to converge to local minima rather than saddle points in a networked setting?
• RQ5 To what extent can the theoretical guarantees of centralized SGD be extended to distributed, network-based first-order algorithms?

Key findings

• For nonconvex and locally Lipschitz continuous loss functions, D-SGD converges to critical points of the global objective function with probability one.
• When the loss functions are smooth in a neighborhood of a saddle point and the saddle is regular, D-SGD avoids such saddle points with probability one.
• The convergence to critical points holds even when individual agent functions are nonsmooth, extending prior results that required convex or difference-of-convex regularization.
• Saddle point avoidance is established via a novel application of stable manifold theory to nonautonomous systems, which is not covered by classical results.
• The analysis relies on approximating D-SGD with a continuous-time distributed gradient flow, and the convergence is proven using measure-theoretic arguments on the probability of escaping neighborhoods of saddle points.
• The results are valid in the general framework of subspace-constrained optimization, making them applicable to a wide range of distributed learning problems, including federated learning and decentralized empirical risk minimization.