[Paper Review] Variance-Reduced Proximal Stochastic Gradient Descent for Non-convex Composite optimization

Xiyu Yu, Dacheng Tao|arXiv (Cornell University)|Jun 2, 2016

Stochastic Gradient Optimization Techniques10 references3 citations

TL;DR

This paper proposes variance-reduced proximal stochastic gradient methods—prox-SVRG and prox-SAGA—for non-convex composite optimization, where the objective is a sum of smooth non-convex functions and a general regularizer with a simple proximal mapping. With constant step size, both methods converge to a stationary point in $O(1/\varepsilon)$ iterations, matching the state-of-the-art RSAG method and outperforming standard stochastic gradient descent.

ABSTRACT

Here we study non-convex composite optimization: first, a finite-sum of smooth but non-convex functions, and second, a general function that admits a simple proximal mapping. Most research on stochastic methods for composite optimization assumes convexity or strong convexity of each function. In this paper, we extend this problem into the non-convex setting using variance reduction techniques, such as prox-SVRG and prox-SAGA. We prove that, with a constant step size, both prox-SVRG and prox-SAGA are suitable for non-convex composite optimization, and help the problem converge to a stationary point within $O(1/ε)$ iterations. That is similar to the convergence rate seen with the state-of-the-art RSAG method and faster than stochastic gradient descent. Our analysis is also extended into the min-batch setting, which linearly accelerates the convergence. To the best of our knowledge, this is the first analysis of convergence rate of variance-reduced proximal stochastic gradient for non-convex composite optimization.

Motivation & Objective

• To extend variance-reduced proximal stochastic gradient methods to non-convex composite optimization problems.
• To analyze the convergence rate of prox-SVRG and prox-SAGA under non-convexity assumptions.
• To establish that these methods converge to a stationary point in $O(1/\varepsilon)$ iterations with constant step size.
• To extend the analysis to the min-batch setting for linear acceleration of convergence.
• To provide the first theoretical convergence rate guarantee for variance-reduced proximal stochastic gradient methods in the non-convex composite setting.

Proposed method

• Adapt the prox-SVRG and prox-SAGA algorithms to handle non-convex finite-sum components and a general regularizer with a simple proximal mapping.
• Use variance reduction techniques to stabilize gradient estimates and improve convergence in non-convex settings.
• Employ a constant step size to ensure convergence to a stationary point within $O(1/\varepsilon)$ iterations.
• Extend the framework to a min-batch setting, achieving linear acceleration of convergence.
• Leverage existing proximal gradient frameworks and integrate them with variance reduction mechanisms for non-convex objectives.
• Prove convergence to a stationary point by analyzing the expected decrease in objective function value per iteration.

Experimental results

Research questions

• RQ1 Can prox-SVRG and prox-SAGA be effectively extended to non-convex composite optimization problems?
• RQ2 What is the convergence rate of variance-reduced proximal stochastic gradient methods in the non-convex setting?
• RQ3 Does the use of constant step size still ensure convergence to a stationary point in non-convex composite problems?
• RQ4 How does the min-batch variant improve convergence speed compared to the single-sample version?
• RQ5 Is the convergence rate of these methods comparable to the state-of-the-art RSAG method in non-convex settings?

Key findings

• The proposed prox-SVRG and prox-SAGA methods converge to a stationary point in $O(1/\varepsilon)$ iterations for non-convex composite optimization with a constant step size.
• The convergence rate matches that of the state-of-the-art RSAG method, outperforming standard stochastic gradient descent.
• The min-batch variant linearly accelerates convergence, improving scalability and efficiency.
• The analysis establishes the first theoretical convergence rate guarantee for variance-reduced proximal stochastic gradient methods in the non-convex composite setting.
• The methods are effective even when individual components are non-convex, provided the regularizer admits a simple proximal mapping.