[Paper Review] Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optima

Po‐Ling Loh, Martin J. Wainwright|arXiv (Cornell University)|May 10, 2013

Sparse and Compressive Sensing TechniquesEngineering17 references185 citations

TL;DR

This paper establishes that all local optima of regularized M-estimators with nonconvex loss and penalty functions lie within statistical precision of the true parameter vector under restricted strong convexity and regularity conditions. It proves that standard first-order methods like composite gradient descent converge to these well-behaved local optima in logarithmic number of steps, eliminating the need for specialized global optimization algorithms.

ABSTRACT

We provide novel theoretical results regarding local optima of regularized $M$-estimators, allowing for nonconvexity in both loss and penalty functions. Under restricted strong convexity on the loss and suitable regularity conditions on the penalty, we prove that \emph{any stationary point} of the composite objective function will lie within statistical precision of the underlying parameter vector. Our theory covers many nonconvex objective functions of interest, including the corrected Lasso for errors-in-variables linear models; regression for generalized linear models with nonconvex penalties such as SCAD, MCP, and capped-$\ell_1$; and high-dimensional graphical model estimation. We quantify statistical accuracy by providing bounds on the $\ell_1$-, $\ell_2$-, and prediction error between stationary points and the population-level optimum. We also propose a simple modification of composite gradient descent that may be used to obtain a near-global optimum within statistical precision $ε$ in $\log(1/ε)$ steps, which is the fastest possible rate of any first-order method. We provide simulation studies illustrating the sharpness of our theoretical results.

Motivation & Objective

• To close the gap between statistical theory and practice in high-dimensional nonconvex M-estimation, where global optima are often computationally infeasible.
• To establish that local optima of nonconvex regularized M-estimators are statistically as good as global optima under mild regularity conditions.
• To provide theoretical guarantees for standard first-order optimization methods to converge to statistically optimal solutions without requiring global optimization.
• To unify and extend prior results on nonconvex penalties like SCAD, MCP, and capped-ℓ₁ in high-dimensional statistical models.
• To demonstrate that stationary points of composite objectives are within statistical error of the population parameter, even when the objective is nonconvex.

Proposed method

• Introduces a general framework for analyzing local optima of regularized M-estimators with nonconvex loss and penalty functions.
• Employs restricted strong convexity on the loss function and regularity conditions on the penalty to bound the distance between any stationary point and the true parameter.
• Uses a convex upper bound on the nonconvex penalty function to derive first-order optimality conditions and error bounds.
• Applies the theory to specific models including corrected Lasso, generalized linear models with SCAD/MCP/capped-ℓ₁ penalties, and high-dimensional graphical models.
• Proposes a modified composite gradient descent algorithm that converges to a solution within statistical precision ε_stat in O(log(1/ε_stat)) steps.
• Leverages decomposability and subgradient inequalities to bound ℓ₁, ℓ₂, and prediction errors between stationary points and the true parameter vector.

Experimental results

Research questions

• RQ1 Under what conditions do all local optima of nonconvex regularized M-estimators lie within statistical error of the true parameter?
• RQ2 Can standard first-order optimization methods like composite gradient descent converge to a solution that is statistically optimal, even when the objective is nonconvex?
• RQ3 How do nonconvex penalties like SCAD, MCP, and capped-ℓ₁ affect the statistical and optimization error of stationary points?
• RQ4 Is it possible to guarantee that any stationary point of a nonconvex M-estimator is as good as a global optimum from a statistical perspective?
• RQ5 What modifications to first-order methods ensure convergence to a solution within statistical precision in the fastest possible rate?

Key findings

• Any stationary point of the regularized M-estimator lies within ℓ₂, ℓ₁, and prediction error bounds that scale with the statistical precision ε_stat, under restricted strong convexity and regularity conditions.
• The modified composite gradient descent algorithm converges to a solution within ε_stat of the true parameter in O(log(1/ε_stat)) iterations, achieving the fastest possible rate for first-order methods.
• For the capped-ℓ₁ penalty with parameter c, the theory shows that the regularization satisfies the required conditions with μ₂ = 1/c, ensuring well-behaved local optima.
• The results subsume previous work on the corrected Lasso and extend to generalized linear models with nonconvex penalties, showing that local optima are statistically consistent.
• The analysis confirms that local optima are not only computationally accessible but also statistically optimal, resolving a key gap between theory and practice in high-dimensional statistics.
• The paper establishes that standard first-order methods can achieve statistical accuracy without requiring specialized algorithms to target specific local minima.