[Paper Review] Deep Sparse Coding Using Optimized Linear Expansion of Thresholds

Debabrata Mahapatra, Subhadip Mukherjee+1 more|arXiv (Cornell University)|May 20, 2017

Sparse and Compressive Sensing Techniques37 references13 citations

TL;DR

This paper proposes a deep neural network (DNN) for sparse signal recovery from compressive, noisy measurements by unfolding the FISTA algorithm with learnable activation functions based on the Linear Expansion of Thresholds (LET). The network uses Hessian-free optimization to train only five parameters per layer, achieving 3–4 dB higher reconstruction SNR than state-of-the-art sparse coding methods, with performance comparable to residual networks despite half the layers.

ABSTRACT

We address the problem of reconstructing sparse signals from noisy and compressive measurements using a feed-forward deep neural network (DNN) with an architecture motivated by the iterative shrinkage-thresholding algorithm (ISTA). We maintain the weights and biases of the network links as prescribed by ISTA and model the nonlinear activation function using a linear expansion of thresholds (LET), which has been very successful in image denoising and deconvolution. The optimal set of coefficients of the parametrized activation is learned over a training dataset containing measurement-sparse signal pairs, corresponding to a fixed sensing matrix. For training, we develop an efficient second-order algorithm, which requires only matrix-vector product computations in every training epoch (Hessian-free optimization) and offers superior convergence performance than gradient-descent optimization. Subsequently, we derive an improved network architecture inspired by FISTA, a faster version of ISTA, to achieve similar signal estimation performance with about 50% of the number of layers. The resulting architecture turns out to be a deep residual network, which has recently been shown to exhibit superior performance in several visual recognition tasks. Numerical experiments demonstrate that the proposed DNN architectures lead to 3 to 4 dB improvement in the reconstruction signal-to-noise ratio (SNR), compared with the state-of-the-art sparse coding algorithms.

Motivation & Objective

• To improve sparse signal reconstruction from noisy, compressive measurements using deep learning.
• To develop a DNN architecture that mirrors iterative shrinkage algorithms like FISTA, with learnable nonlinearities for better performance.
• To enable efficient training via Hessian-free optimization, avoiding explicit Hessian computation.
• To reduce the number of layers while maintaining performance, leading to a residual-like network structure.
• To achieve superior reconstruction SNR compared to classical sparse coding and state-of-the-art methods.

Proposed method

• The network architecture is derived by unfolding the FISTA algorithm, with fixed weights and biases per layer as in ISTA/FISTA.
• Nonlinear activation functions are modeled using a parametrized Linear Expansion of Thresholds (LET), initialized with soft-thresholding.
• The optimal LET coefficients are learned via a custom Hessian-free optimization algorithm that computes only Hessian-vector products.
• The training process uses second-order optimization with minimal memory overhead, enabling fast convergence.
• The resulting network is a deep residual network (DRN) variant, achieving high performance with approximately 50% fewer layers than standard FISTA-based DNNs.
• The method is trained on measurement-sparse signal pairs for a fixed sensing matrix, with performance evaluated via SNR and support recovery metrics.

Experimental results

Research questions

• RQ1 Can a deep neural network trained via Hessian-free optimization outperform classical sparse coding algorithms in signal reconstruction from compressive measurements?
• RQ2 Can a learnable activation function based on the Linear Expansion of Thresholds (LET) improve sparse recovery performance beyond standard soft-thresholding?
• RQ3 Does unfolding FISTA into a DNN with shared weights and learnable LET activations yield better reconstruction SNR with fewer layers than standard DNNs?
• RQ4 To what extent does tying parameters across layers affect performance, and can the network still outperform state-of-the-art methods?
• RQ5 How does the network's support stabilization behavior correlate with SNR improvement during inference?

Key findings

• The proposed fLETnet architecture achieves a 3–4 dB improvement in reconstruction signal-to-noise ratio (SNR) compared to state-of-the-art sparse coding algorithms.
• The network achieves performance comparable to a full residual network (DRN) with approximately 50% fewer layers, indicating high sample efficiency.
• Hessian-free optimization enables faster convergence and superior training performance compared to standard gradient descent.
• Tying the LET parameters across layers results in only a 2 dB SNR degradation, still outperforming competing methods by 2–3 dB.
• The network stabilizes support estimation early in inference, after which it primarily performs amplitude regression, as confirmed by SRM analysis.
• The method discovers layer-specific regularizers through learning, which collectively outperform standard ℓ₁ regularization in sparse recovery.