[Paper Review] Consistent Parameter Estimation for LASSO and Approximate Message Passing

Ali Mousavi, Arian Maleki+1 more|arXiv (Cornell University)|Nov 3, 2015

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper provides an asymptotic analysis of LASSO and Approximate Message Passing (AMP), showing that the active set size and mean squared error (MSE) of their solutions vary predictably with the regularization parameter λ or threshold parameters τᵗ. It proposes a computationally efficient, asymptotically consistent tuning method for AMP that achieves optimal MSE equivalent to the best-tuned LASSO solution.

ABSTRACT

We consider the problem of recovering a vector $β_o \in \mathbb{R}^p$ from $n$ random and noisy linear observations $y= Xβ_o + w$, where $X$ is the measurement matrix and $w$ is noise. The LASSO estimate is given by the solution to the optimization problem $\hatβ_λ = \arg \min_β \frac{1}{2} \|y-Xβ\|_2^2 + λ\| β\|_1$. Among the iterative algorithms that have been proposed for solving this optimization problem, approximate message passing (AMP) has attracted attention for its fast convergence. Despite significant progress in the theoretical analysis of the estimates of LASSO and AMP, little is known about their behavior as a function of the regularization parameter $λ$, or the thereshold parameters $τ^t$. For instance the following basic questions have not yet been studied in the literature: (i) How does the size of the active set $\|\hatβ^λ\|_0/p$ behave as a function of $λ$? (ii) How does the mean square error $\|\hatβ_λ - β_o\|_2^2/p$ behave as a function of $λ$? (iii) How does $\|β^t - β_o \|_2^2/p$ behave as a function of $τ^1, \ldots, τ^{t-1}$? Answering these questions will help in addressing practical challenges regarding the optimal tuning of $λ$ or $τ^1, τ^2, \ldots$. This paper answers these questions in the asymptotic setting and shows how these results can be employed in deriving simple and theoretically optimal approaches for tuning the parameters $τ^1, \ldots, τ^t$ for AMP or $λ$ for LASSO. It also explores the connection between the optimal tuning of the parameters of AMP and the optimal tuning of LASSO.

Motivation & Objective

• To understand how the size of the active set and mean squared error (MSE) of LASSO and AMP solutions depend on the regularization parameter λ or threshold parameters τᵗ.
• To address the lack of theoretical understanding of parameter tuning behavior in LASSO and AMP, especially in high-dimensional asymptotic regimes.
• To develop a computationally efficient and asymptotically consistent parameter tuning method for AMP that matches the optimal performance of LASSO.
• To establish a theoretical connection between optimal tuning of AMP and optimal tuning of LASSO in the asymptotic limit.

Proposed method

• Analyzes the asymptotic behavior of LASSO and AMP in the limit where p → ∞ and n/p → δ ∈ (0,1), using random matrix theory and asymptotic statistical analysis.
• Derives the functional dependence of the active set size ‖β̂λ‖₀/p on λ and the MSE ‖β̂λ − βₒ‖₂²/p on λ, showing quasi-convexity of MSE in λ.
• Proposes a modified bisection method to tune AMP’s threshold parameters τ¹, τ², ..., τᵗ iteratively, based on the asymptotic behavior of MSE.
• Uses the state evolution framework to model the dynamics of AMP and derives a consistent tuning rule that minimizes MSE asymptotically.
• Validates the proposed tuning scheme through simulations comparing convergence speed and reconstruction accuracy against grid search, maximin, and constant threshold methods.
• Demonstrates that the optimal AMP solution under the proposed tuning achieves the same MSE as the optimal LASSO solution with the best λ.

Experimental results

Research questions

• RQ1 How does the size of the active set ‖β̂λ‖₀/p behave as a function of the regularization parameter λ in the LASSO estimator?
• RQ2 How does the mean squared error ‖β̂λ − βₒ‖₂²/p vary with λ, and is it quasi-convex as suggested by the analysis?
• RQ3 How does the estimation error ‖βᵗ − βₒ‖₂²/p in AMP evolve as a function of the sequence of threshold parameters τ¹, ..., τᵗ?
• RQ4 Can a consistent and computationally efficient tuning scheme be derived for AMP’s threshold parameters τᵗ that matches the optimal LASSO performance?
• RQ5 What is the theoretical connection between the optimal tuning of AMP and the optimal tuning of LASSO in the high-dimensional asymptotic regime?

Key findings

• The active set size ‖β̂λ‖₀/p is a decreasing function of λ, consistent with intuitive expectations from LASSO regularization.
• The mean squared error ‖β̂λ − βₒ‖₂²/p is a quasi-convex function of λ, implying the existence of a unique optimal λ that minimizes MSE.
• The proposed bisection-based tuning scheme for AMP parameters τᵗ achieves faster convergence and lower reconstruction error than maximin or constant threshold tuning.
• Parameterless AMP using the proposed method achieves the same optimal MSE as the best-tuned LASSO solution, demonstrating theoretical equivalence in performance.
• The proposed tuning scheme requires significantly fewer computations than grid search, while outperforming both grid search and maximin approaches in convergence speed and accuracy.
• The method adapts to signal sparsity and noise variance, achieving optimal MSE across different signal and noise distributions, unlike maximin AMP which is only optimal in noiseless settings.