[Paper Review] SAGA and Restricted Strong Convexity

Chao Qu, Yan Li+1 more|arXiv (Cornell University)|Feb 19, 2017

Sparse and Compressive Sensing Techniques28 references3 citations

TL;DR

This paper establishes linear convergence of the SAGA algorithm for non-strongly convex and non-convex statistical problems—such as Lasso, group Lasso, logistic regression with ℓ₁ regularization, corrected Lasso, and SCAD-regularized regression—under the restricted strong convexity (RSC) condition. It proves that SAGA converges linearly up to the statistical estimation accuracy, significantly extending its applicability beyond strongly convex settings.

ABSTRACT

SAGA is a fast incremental gradient method on the finite sum problem and its effectiveness has been tested on a vast of applications. In this paper, we analyze SAGA on a class of non-strongly convex and non-convex statistical problem such as Lasso, group Lasso, Logistic regression with $\ell_1$ regularization, linear regression with SCAD regularization and Correct Lasso. We prove that SAGA enjoys the linear convergence rate up to the statistical estimation accuracy, under the assumption of restricted strong convexity (RSC). It significantly extends the applicability of SAGA in convex and non-convex optimization.

Motivation & Objective

• To extend the convergence analysis of SAGA beyond strongly convex problems to non-convex and non-strongly convex statistical models.
• To establish linear convergence of SAGA up to the statistical estimation error under the restricted strong convexity (RSC) condition.
• To analyze SAGA's performance on key machine learning problems including Lasso, group Lasso, logistic regression with ℓ₁ regularization, corrected Lasso, and SCAD-regularized regression.
• To provide theoretical justification for SAGA's empirical success in high-dimensional and non-convex settings.

Proposed method

• Theoretical analysis of SAGA under the restricted strong convexity (RSC) condition, which relaxes strong convexity by requiring curvature only on a subspace of interest.
• Use of the proximal SAGA update rule with an unbiased gradient estimator to maintain convergence properties while avoiding full gradient computation.
• Application of RSC to control the curvature of the objective function in the direction of the true sparse parameter, ensuring linear convergence to the statistical precision.
• Bounding the dual norm of the gradient at the optimum using tail bounds on Gaussian noise, ensuring the regularization parameter λ satisfies necessary conditions.
• Derivation of tolerance bounds δ that scale with the statistical error, showing convergence to within a neighborhood of the true parameter.
• Leveraging subspace compatibility constants and norm compatibility to quantify convergence rates under RSC.

Experimental results

Research questions

• RQ1 Does SAGA achieve linear convergence for non-strongly convex problems such as Lasso and group Lasso under the RSC condition?
• RQ2 Can SAGA maintain linear convergence in non-convex settings like SCAD and corrected Lasso regularization?
• RQ3 What is the convergence rate of SAGA when the objective function is neither strongly convex nor convex, but satisfies RSC?
• RQ4 How does the regularization parameter λ relate to the noise level and data dimension in high-dimensional models?
• RQ5 What is the statistical accuracy threshold beyond which SAGA converges linearly?

Key findings

• SAGA achieves linear convergence up to the statistical estimation accuracy under the restricted strong convexity (RSC) condition, even for non-convex and non-strongly convex problems.
• For Lasso, the tolerance δ scales as c₂ν(Σ)(r log p)/n × ||Δ*||₂², with high probability under Gaussian noise.
• In group Lasso, the tolerance δ depends on the group structure and scales as c₂σ₂(Σ)s_G(√(m/n) + √(3 log N_G / n))² × ||Δ*||₂².
• For corrected Lasso, the gradient norm at the optimum is bounded by c₁φ√(log p / n), with high probability, where φ depends on Σ and θ*.
• The RSC condition holds with high probability for i.i.d. sub-Gaussian design matrices, enabling convergence guarantees in high-dimensional settings.
• The analysis confirms that SAGA's convergence is robust to model misspecification and noise, achieving linear rates near the statistical limit.