[Paper Review] Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds

Xiaohan Chen, Jialin Liu+2 more|arXiv (Cornell University)|Aug 29, 2018

Optical Systems and Laser Technology152 citations

TL;DR

The paper proves linear convergence for unfolded ISTA (LISTA) under a partial weight coupling, introduces a support-selection technique to enhance convergence, and validates the theory with simulations and a natural image CS experiment.

ABSTRACT

In recent years, unfolding iterative algorithms as neural networks has become an empirical success in solving sparse recovery problems. However, its theoretical understanding is still immature, which prevents us from fully utilizing the power of neural networks. In this work, we study unfolded ISTA (Iterative Shrinkage Thresholding Algorithm) for sparse signal recovery. We introduce a weight structure that is necessary for asymptotic convergence to the true sparse signal. With this structure, unfolded ISTA can attain a linear convergence, which is better than the sublinear convergence of ISTA/FISTA in general cases. Furthermore, we propose to incorporate thresholding in the network to perform support selection, which is easy to implement and able to boost the convergence rate both theoretically and empirically. Extensive simulations, including sparse vector recovery and a compressive sensing experiment on real image data, corroborate our theoretical results and demonstrate their practical usefulness. We have made our codes publicly available: https://github.com/xchen-tamu/linear-lista-cpss.

Motivation & Objective

• Motivate improving sparse recovery by unfolding ISTA into a trainable network (LISTA).
• Identify structural conditions on LISTA weights that guarantee linear convergence to the true sparse signal.
• Propose a practical support-selection mechanism to boost convergence rate and recovery accuracy.
• Demonstrate both theoretically and empirically that the proposed LISTA variants outperform baselines in noisy and noiseless settings.

Proposed method

• Introduce a partial weight coupling structure W2 ≈ I − W1A and show it is asymptotically necessary for convergence (Theorem 1).
• Derive a simplified LISTA update with coupled weights: x^{k+1} = η_{θ^k}(x^k + W^{k T}(b − A x^k)).
• Propose LISTA-CP (with weight coupling) and prove linear convergence in the noiseless case (Theorem 2).
• Enhance LISTA with a support-selection (SS) mechanism that protects top-magnitude entries from thresholding, leading to LISTA-SS and LISTA-CPSS (Theorem 3).
• Provide convergence bounds: linear rate bounds in noiseless and noisy settings, and improved rates with SS under mild assumptions.

Experimental results

Research questions

• RQ1 Can LISTA be guaranteed to converge linearly to the true sparse signal under learnable parameters?
• RQ2 What weight structures and parameterizations enable faster convergence than standard ISTA/FISTA?
• RQ3 Does incorporating a threshold-based support selection boost convergence rate and recovery accuracy?
• RQ4 How do the proposed LISTA variants perform in noiseless and noisy regimes, and under ill-conditioned dictionaries?

Key findings

• LISTA-2 (with partial weight coupling) has a necessary condition W2 → I − W1A and θ^k → 0 for convergence (Theorem 1).
• With weight coupling, LISTA simplifies to a two-parameter trainable form and empirically matches or exceeds baseline LISTA performance (LISTA-CP results).
• There exists a sequence of trainable parameters yielding linear convergence from the first iteration (Theorem 2).
• Thresholding with a learnable path of thresholds (adaptive λ-like behavior) improves convergence and accuracy over ISTA in simulations.
• Thresholded support selection (LISTA-CPSS) achieves faster convergence rates and lower steady-state error than LISTA-CP, especially as layer depth grows (Theorem 3).
• For ill-conditioned A, LISTA-CP/CPSS maintains superior performance over LISTA and LAMP across varying condition numbers.