[Paper Review] Asymptotic errors for convex penalized linear regression beyond Gaussian matrices

Cédric Gerbelot, Alia Abbara+1 more|arXiv (Cornell University)|Feb 11, 2020

Sparse and Compressive Sensing Techniques65 references18 citations

TL;DR

This paper provides a rigorous derivation of asymptotic mean squared error formulas for convex penalized linear regression—such as LASSO and elastic net—beyond i.i.d. Gaussian matrices, extending previous results to rotationally invariant random matrices with arbitrary singular value spectra. The analysis leverages an oracle version of vector approximate message-passing (oracle-VAMP) and its state evolution, establishing a formal link between proximal descent algorithms and message-passing frameworks, with strong agreement between asymptotic predictions and finite-size simulations.

ABSTRACT

We consider the problem of learning a coefficient vector $x_{0}$ in $R^{N}$ from noisy linear observations $y=Fx_{0}+w$ in $R^{M}$ in the high dimensional limit $M,N$ to infinity with $α=M/N$ fixed. We provide a rigorous derivation of an explicit formula -- first conjectured using heuristic methods from statistical physics -- for the asymptotic mean squared error obtained by penalized convex regression estimators such as the LASSO or the elastic net, for a class of very generic random matrices corresponding to rotationally invariant data matrices with arbitrary spectrum. The proof is based on a convergence analysis of an oracle version of vector approximate message-passing (oracle-VAMP) and on the properties of its state evolution equations. Our method leverages on and highlights the link between vector approximate message-passing, Douglas-Rachford splitting and proximal descent algorithms, extending previous results obtained with i.i.d. matrices for a large class of problems. We illustrate our results on some concrete examples and show that even though they are asymptotic, our predictions agree remarkably well with numerics even for very moderate sizes.

Motivation & Objective

• To derive an explicit, asymptotically exact formula for the mean squared error (MSE) in convex penalized linear regression under high-dimensional limits.
• To extend prior results—previously limited to i.i.d. Gaussian matrices—beyond the Gaussian assumption to a broad class of rotationally invariant random matrices with arbitrary spectra.
• To provide a mathematically rigorous proof of the replica formula for reconstruction error, previously derived heuristically using statistical physics methods.
• To establish a formal connection between proximal descent algorithms, Douglas-Rachford splitting, and message-passing algorithms such as VAMP.
• To demonstrate convergence guarantees for an oracle version of VAMP and propose a method to enforce convergence via regularization.

Proposed method

• The analysis is based on the convergence properties of an oracle version of vector approximate message-passing (oracle-VAMP), which is designed to handle structured random matrices.
• State evolution equations for oracle-VAMP are derived and analyzed to track the asymptotic behavior of the estimation error in high-dimensional settings.
• The method maps proximal operators used in convex optimization to denoising functions in message-passing algorithms, enabling a rigorous link between optimization and inference frameworks.
• The proof relies on pseudo-Lipschitz continuity and empirical convergence of vector sequences, ensuring almost-sure convergence of key statistics in the limit N,M→∞.
• The framework is validated by showing that the state evolution equations converge to a fixed point that corresponds to the predicted MSE.
• A regularization parameter (ridge term) is introduced to stabilize the algorithm and enforce convergence in ill-conditioned regimes.

Experimental results

Research questions

• RQ1 What is the asymptotic mean squared error of convex penalized regression estimators when the design matrix is rotationally invariant with arbitrary singular values, rather than i.i.d. Gaussian?
• RQ2 Can the replica formula for reconstruction error—previously derived heuristically via statistical physics—be rigorously proven for generic rotationally invariant matrices?
• RQ3 How do proximal descent algorithms relate to message-passing algorithms such as VAMP in the high-dimensional limit?
• RQ4 What conditions ensure the convergence of oracle-VAMP for non-Gaussian, structured random matrices?
• RQ5 To what extent do the asymptotic predictions hold in finite-size settings, and how can convergence be enforced in ill-conditioned scenarios?

Key findings

• The paper derives an explicit, asymptotically exact formula for the mean squared error of convex penalized regression estimators, valid for all rotationally invariant matrices with arbitrary spectra.
• The derived formula rigorously confirms the replica formula previously conjectured via statistical physics methods, marking the first such proof for generic rotationally invariant matrices.
• The convergence of oracle-VAMP is proven under mild conditions, with convergence rates bounded by the properties of the proximal operators and the matrix spectrum.
• Numerical results show excellent agreement between the asymptotic predictions and simulations even for moderate system sizes (N=100), validating the robustness of the asymptotic analysis.
• Increasing the ridge parameter (λ₂) in elastic net problems effectively stabilizes the algorithm and enables convergence in highly ill-conditioned regimes (e.g., low aspect ratio α=0.1).
• The framework establishes a formal equivalence between proximal descent, Douglas-Rachford splitting, and maximum a posteriori message-passing, deepening the theoretical understanding of these algorithms.