[Paper Review] Maximal Sparsity with Deep Networks?

Bo Xin, Yizhou Wang+2 more|arXiv (Cornell University)|May 5, 2016

Sparse and Compressive Sensing Techniques37 references96 citations

TL;DR

This paper proposes a deep learning approach that outperforms traditional sparse estimation algorithms in recovering maximally sparse representations, especially when the dictionary has high coherence (large restricted isometry constant). By learning layer weights through end-to-end training instead of using fixed algorithmic updates, the network achieves superior accuracy in sparse recovery, even in regimes where standard methods fail, as validated on a photometric stereo problem with sparse outliers.

ABSTRACT

The iterations of many sparse estimation algorithms are comprised of a fixed linear filter cascaded with a thresholding nonlinearity, which collectively resemble a typical neural network layer. Consequently, a lengthy sequence of algorithm iterations can be viewed as a deep network with shared, hand-crafted layer weights. It is therefore quite natural to examine the degree to which a learned network model might act as a viable surrogate for traditional sparse estimation in domains where ample training data is available. While the possibility of a reduced computational budget is readily apparent when a ceiling is imposed on the number of layers, our work primarily focuses on estimation accuracy. In particular, it is well-known that when a signal dictionary has coherent columns, as quantified by a large RIP constant, then most tractable iterative algorithms are unable to find maximally sparse representations. In contrast, we demonstrate both theoretically and empirically the potential for a trained deep network to recover minimal $\ell_0$-norm representations in regimes where existing methods fail. The resulting system is deployed on a practical photometric stereo estimation problem, where the goal is to remove sparse outliers that can disrupt the estimation of surface normals from a 3D scene.

Motivation & Objective

• To address the limitation of traditional sparse estimation algorithms in recovering maximally sparse solutions when the dictionary has high coherence, quantified by large restricted isometry constants.
• To investigate whether a deep neural network with learned weights can serve as a superior surrogate for iterative sparse recovery algorithms in high-coherence regimes.
• To demonstrate that end-to-end trained deep networks can achieve exact support recovery and minimal ℓ₀-norm solutions where standard methods fail.
• To validate the approach on a real-world photometric stereo problem involving robust surface normal estimation under sparse outlier corruption.

Proposed method

• The method reformulates iterative sparse recovery algorithms as deep networks with shared, hand-crafted weights, then replaces these with discriminatively trained, layer-specific weights.
• The network architecture is designed as a deep unfolding of iterative hard thresholding (IHT), with learnable filters and thresholding components.
• Support templates are dynamically updated based on estimated supports, enabling partial support recovery and improved convergence.
• The network uses a modified IHT update rule with a collapsed dictionary formed from cluster centers and active components, improving stability and sparsity.
• A two-stage process is employed: first, cluster centers are identified; second, the remaining components are recovered using known support information.
• The training process uses paired data of sparse signals and their observations to optimize for minimal ℓ₀-norm solutions.

Experimental results

Research questions

• RQ1 Can a learned deep network achieve exact support recovery in sparse representation problems where traditional algorithms fail due to high dictionary coherence?
• RQ2 Does end-to-end training of a deep network with learned weights significantly improve estimation accuracy compared to fixed-weights iterative algorithms?
• RQ3 To what extent can a deep network recover maximally sparse solutions (minimal ℓ₀-norm) in the presence of high-restricted isometry constants?
• RQ4 How does the proposed method perform in practical applications involving sparse outliers, such as photometric stereo?

Key findings

• The proposed deep network achieves exact support recovery and minimal ℓ₀-norm solutions in regimes where standard iterative methods fail due to high coherence in the dictionary.
• The network converges to the true sparse solution after a finite number of iterations, with reconstruction error decaying exponentially as 2⁻ᵗ.
• Theoretical analysis shows that with known partial support, the required restricted isometry constant for unique recovery is significantly relaxed, enabling convergence even under high coherence.
• Empirical results on photometric stereo demonstrate that the method effectively removes sparse outliers and improves surface normal estimation accuracy.
• The network outperforms traditional sparse recovery algorithms in both accuracy and robustness, particularly when the restricted isometry constant exceeds the theoretical threshold for standard methods.