[Paper Review] Distributed Projected Subgradient Method for Weakly Convex Optimization.

Shixiang Chen, Alfredo García+1 more|arXiv (Cornell University)|Apr 28, 2020

Sparse and Compressive Sensing Techniques30 references4 citations

TL;DR

This paper proposes a distributed projected stochastic subgradient method (stoDPSM) for weakly convex optimization, proving its global convergence via the Moreau envelope stationarity measure. Under a sharpness condition, deterministic DPSM with geometrically diminishing step-sizes converges linearly to sharp minima, offering theoretical guarantees for large-scale machine learning problems like dictionary learning and robust phase retrieval.

ABSTRACT

The stochastic subgradient method is a widely-used algorithm for solving large-scale optimization problems arising in machine learning. Often these problems are neither smooth nor convex. Recently, Davis et al. [1-2] characterized the convergence of the stochastic subgradient method for the weakly convex case, which encompasses many important applications (e.g., robust phase retrieval, blind deconvolution, biconvex compressive sensing, and dictionary learning). In practice, distributed implementations of the projected stochastic subgradient method (stoDPSM) are used to speed-up risk minimization. In this paper, we propose a distributed implementation of the stochastic subgradient method with a theoretical guarantee. Specifically, we show the global convergence of stoDPSM using the Moreau envelope stationarity measure. Furthermore, under a so-called sharpness condition, we show that deterministic DPSM (with a proper initialization) converges linearly to the sharp minima, using geometrically diminishing step-size. We provide numerical experiments to support our theoretical analysis.

Motivation & Objective

• To address the lack of theoretical convergence guarantees for distributed implementations of stochastic subgradient methods in non-convex, non-smooth settings.
• To extend the convergence analysis of stochastic subgradient methods to weakly convex problems, which include key machine learning applications.
• To establish global convergence of the distributed projected stochastic subgradient method (stoDPSM) using the Moreau envelope stationarity measure.
• To prove linear convergence of deterministic DPSM under a sharpness condition with geometrically diminishing step-sizes.
• To validate theoretical findings with numerical experiments on weakly convex optimization tasks.

Proposed method

• Proposes a distributed projected stochastic subgradient method (stoDPSM) for weakly convex optimization, where each agent updates using local subgradients and projections.
• Employs the Moreau envelope stationarity measure as a convergence criterion, which is suitable for non-smooth and weakly convex problems.
• Introduces a geometrically diminishing step-size rule in the deterministic DPSM variant to achieve linear convergence under the sharpness condition.
• Requires proper initialization to ensure convergence to sharp minima in the deterministic setting.
• Uses distributed computation with local updates and consensus mechanisms to scale optimization across multiple agents.
• Analyzes convergence under weak convexity by leveraging properties of the Moreau envelope and subgradient dynamics.

Experimental results

Research questions

• RQ1 Does the distributed projected stochastic subgradient method (stoDPSM) converge globally for weakly convex optimization problems?
• RQ2 Can the Moreau envelope stationarity measure serve as a valid convergence criterion for non-smooth, weakly convex problems in distributed settings?
• RQ3 Under what conditions does deterministic DPSM achieve linear convergence to sharp minima?
• RQ4 How does geometrically diminishing step-size impact convergence in the deterministic DPSM under sharpness?
• RQ5 Do numerical experiments confirm the theoretical convergence behavior of the proposed distributed methods?

Key findings

• The proposed stoDPSM achieves global convergence for weakly convex optimization problems, as measured by the Moreau envelope stationarity.
• Under the sharpness condition, deterministic DPSM with geometrically diminishing step-sizes converges linearly to sharp minima.
• The convergence guarantee holds when the initialization is properly chosen, ensuring convergence to the optimal solution in the sharp case.
• The Moreau envelope stationarity measure effectively captures convergence in non-smooth and weakly convex settings, enabling theoretical analysis.
• Numerical experiments support the theoretical findings, demonstrating consistent convergence behavior across test problems.
• The method is applicable to real-world machine learning tasks such as robust phase retrieval, blind deconvolution, and dictionary learning.