[Paper Review] Stochastic Variance Reduced Optimization for Nonconvex Sparse Learning

Xingguo Li, Tuo Zhao+3 more|arXiv (Cornell University)|May 9, 2016

Sparse and Compressive Sensing TechniquesEngineering38 references34 citations

TL;DR

This paper proposes a stochastic variance reduced optimization algorithm for large-scale nonconvex sparse learning with cardinality constraints, achieving strong linear convergence and optimal estimation accuracy in high dimensions. It further introduces an asynchronous variant that delivers near-linear speedup, demonstrating superior computational efficiency and parameter estimation in numerical experiments.

ABSTRACT

We propose a stochastic variance reduced optimization algorithm for solving a class of large-scale nonconvex optimization problems with cardinality constraints, and provide sufficient conditions under which the proposed algorithm enjoys strong linear convergence guarantees and optimal estimation accuracy in high dimensions. We further extend our analysis to an asynchronous variant of the approach, and demonstrate a near linear speedup in sparse settings. Numerical experiments demonstrate the efficiency of our method in terms of both parameter estimation and computational performance.

Motivation & Objective

• To address large-scale nonconvex optimization problems with cardinality constraints in high-dimensional settings.
• To develop an algorithm that ensures strong linear convergence and optimal estimation accuracy despite nonconvexity.
• To extend the method to an asynchronous variant for improved computational scalability in sparse settings.
• To empirically validate the algorithm’s efficiency in both parameter estimation and runtime performance.

Proposed method

• The algorithm employs stochastic variance reduction techniques to stabilize gradient updates in nonconvex sparse optimization.
• It incorporates cardinality constraints directly into the optimization framework to promote sparsity in the solution.
• Theoretical analysis establishes sufficient conditions for strong linear convergence and optimal estimation accuracy in high dimensions.
• An asynchronous variant is designed to enable near-linear speedup in distributed sparse learning environments.
• The method leverages variance-reduced stochastic gradients to improve convergence stability and reduce noise in large-scale settings.

Experimental results

Research questions

• RQ1 Can a stochastic variance reduced algorithm achieve strong linear convergence for nonconvex sparse learning problems?
• RQ2 What conditions ensure optimal estimation accuracy in high-dimensional nonconvex optimization with cardinality constraints?
• RQ3 Does an asynchronous variant of the algorithm achieve near-linear speedup in sparse settings?
• RQ4 How does the proposed method compare to existing approaches in terms of computational efficiency and estimation accuracy?

Key findings

• The proposed algorithm achieves strong linear convergence under sufficient conditions, even in nonconvex and high-dimensional settings.
• Optimal estimation accuracy is guaranteed under the derived theoretical conditions, ensuring statistical consistency.
• The asynchronous variant demonstrates near-linear speedup, confirming scalability in distributed sparse learning.
• Numerical experiments confirm superior performance in both parameter estimation and computational runtime compared to baseline methods.