[Paper Review] A Concise Tutorial on Approximate Message Passing

Qiuyun Zou, Hongwen Yang|arXiv (Cornell University)|Jan 19, 2022

Sparse and Compressive Sensing TechniquesEngineering19 citations

TL;DR

This paper provides a comprehensive tutorial on Approximate Message Passing (AMP) and its advanced variants—OAMP, VAMP, and MAMP—for high-dimensional sparse signal recovery in linear regression. It derives AMP from message passing principles, establishes its state evolution for performance prediction, and extends it to non-i.i.d. measurement matrices via OAMP and MAMP, demonstrating convergence and Bayes-optimal performance under broader conditions.

ABSTRACT

High-dimensional signal recovery of standard linear regression is a key challenge in many engineering fields, such as, communications, compressed sensing, and image processing. The approximate message passing (AMP) algorithm proposed by Donoho extit{et al} is a computational efficient method to such problems, which can attain Bayes-optimal performance in independent identical distributed (IID) sub-Gaussian random matrices region. A significant feature of AMP is that the dynamical behavior of AMP can be fully predicted by a scalar equation termed station evolution (SE). Although AMP is optimal in IID sub-Gaussian random matrices, AMP may fail to converge when measurement matrix is beyond IID sub-Gaussian. To extend the region of random measurement matrix, an expectation propagation (EP)-related algorithm orthogonal AMP (OAMP) was proposed, which shares the same algorithm with EP, expectation consistent (EC), and vector AMP (VAMP). This paper aims at giving a review for those algorithms. We begin with the worst case, i.e., least absolute shrinkage and selection operator (LASSO) inference problem, and then give the detailed derivation of AMP derived from message passing. Also, in the Bayes-optimal setting, we give the Bayes-optimal AMP which has a slight difference from AMP for LASSO. In addition, we review some AMP-related algorithms: OAMP, VAMP, and Memory AMP (MAMP), which can be applied to more general random matrices.

Motivation & Objective

• To provide a unified, accessible tutorial on Approximate Message Passing (AMP) and its advanced variants for researchers in signal processing and machine learning.
• To derive AMP from message passing principles and establish its theoretical foundation using state evolution (SE) for performance prediction.
• To extend AMP’s applicability beyond i.i.d. sub-Gaussian matrices by introducing OAMP and MAMP, which handle more general random measurement matrices.
• To compare the convergence behavior, computational complexity, and mean squared error (MSE) performance of MAMP and OAMP under varying condition numbers.
• To demonstrate that the asymptotic MSE of these algorithms can be fully predicted by their respective state evolution equations.

Proposed method

• Derives AMP from belief propagation and message passing in graphical models, using the Laplace approximation to simplify posterior inference.
• Introduces state evolution (SE) as a deterministic scalar recursion that predicts the asymptotic MSE of AMP in the large-system limit.
• Proposes OAMP as a modification of AMP using a linear minimum mean square error (LMMSE) de-correlated matrix and a divergence-free denoiser to improve convergence for non-i.i.d. matrices.
• Develops MAMP (Memory AMP) by approximating matrix inversion via Taylor series and incorporating memory of past messages to enforce orthogonality and stability.
• Employs spectral decomposition of the measurement matrix to inform MAMP’s design and enables convergence analysis via state evolution.
• Uses numerical simulations with Haar-distributed matrices and varying condition numbers to evaluate convergence speed and NMSE performance of MAMP and OAMP.

Experimental results

Research questions

• RQ1 How can AMP be systematically derived from message passing principles and what is the role of state evolution in predicting its performance?
• RQ2 Why does standard AMP fail to converge for non-i.i.d. measurement matrices, and how can this limitation be overcome?
• RQ3 What are the key design principles of OAMP that allow it to maintain convergence and optimality for unitarily invariant random matrices?
• RQ4 How does MAMP improve upon OAMP in terms of convergence stability and applicability to ill-conditioned matrices, and what is the trade-off in computational complexity?
• RQ5 To what extent can the asymptotic MSE of MAMP and OAMP be predicted by their respective state evolution equations?

Key findings

• AMP achieves Bayes-optimal mean squared error (MSE) performance in the large-system limit when the measurement matrix is i.i.d. sub-Gaussian.
• OAMP extends AMP’s convergence to unitarily invariant random matrices by incorporating an LMMSE de-correlated matrix and a divergence-free denoiser, at the cost of higher computational complexity.
• MAMP achieves stable convergence for a broader class of random matrices by using a memory-based approximation of matrix inversion and enforcing three-way orthogonality across messages.
• In simulations with increasing condition number (κ(H)), both MAMP and OAMP show degraded convergence speed and higher NMSE, but MAMP converges to the same fixed point as OAMP.
• The asymptotic MSE of MAMP is fully predictable via its state evolution equation, which depends on the spectral properties of the measurement matrix and past message statistics.