[Paper Review] Learning step sizes for unfolded sparse coding

Pierre Ablin, Thomas Moreau+2 more|arXiv (Cornell University)|May 27, 2019

Sparse and Compressive Sensing Techniques34 references4 citations

TL;DR

This paper proposes Step-LISTA (SLISTA), a neural network architecture that learns only the step sizes in the Iterative Shrinkage-Thresholding Algorithm (ISTA) for sparse coding, significantly accelerating convergence for sparse solutions. Theoretically, it proves that deep layers of any converging LISTA network reduce to ISTA with learned step sizes, and empirically, SLISTA outperforms state-of-the-art methods in high-sparsity regimes while preserving the dictionary structure.

ABSTRACT

Sparse coding is typically solved by iterative optimization techniques, such as the Iterative Shrinkage-Thresholding Algorithm (ISTA). Unfolding and learning weights of ISTA using neural networks is a practical way to accelerate estimation. In this paper, we study the selection of adapted step sizes for ISTA. We show that a simple step size strategy can improve the convergence rate of ISTA by leveraging the sparsity of the iterates. However, it is impractical in most large-scale applications. Therefore, we propose a network architecture where only the step sizes of ISTA are learned. We demonstrate that for a large class of unfolded algorithms, if the algorithm converges to the solution of the Lasso, its last layers correspond to ISTA with learned step sizes. Experiments show that our method is competitive with state-of-the-art networks when the solutions are sparse enough.

Motivation & Objective

• To accelerate the convergence of ISTA for sparse coding by learning adaptive step sizes.
• To develop a practical neural network architecture that learns only step sizes, avoiding full parameterization.
• To theoretically justify why deep layers of learned ISTA networks must converge to ISTA with learned step sizes.
• To validate that learning only step sizes yields competitive performance with state-of-the-art methods under high sparsity.
• To preserve the dictionary structure, enabling application in structured settings like convolutional or wavelet-based models.

Proposed method

• Proposes Oracle-ISTA, an analytic strategy to compute larger, adaptive step sizes using sparsity-aware Lipschitz constants, improving ISTA's convergence rate.
• Introduces Step-LISTA (SLISTA), a neural network where only the step size parameter α is learned per layer, while the dictionary D is fixed.
• Uses a differentiable, layer-wise update rule: z^(t+1) = ST(z^(t) - α^(t) D^T(Dz^(t) - x), λα^(t)), with α^(t) as the learnable parameter.
• Theoretically proves in Theorem 4.4 that any converging LISTA network's deepest layers must correspond to ISTA with learned step sizes, i.e., W^(t) → D and α^(t)/β^(t) → 1.
• Empirically trains SLISTA on synthetic and semi-real data (e.g., MNIST digits) using an unsupervised loss, minimizing reconstruction error.
• Validates performance via test loss curves across varying regularization parameters λ, comparing against ISTA, LISTA, and ALISTA.

Experimental results

Research questions

• RQ1 Can learning only the step sizes in ISTA significantly improve convergence speed for sparse coding?
• RQ2 What theoretical justification exists for the emergence of learned step sizes in deep learned ISTA networks?
• RQ3 How does SLISTA compare to full LISTA and ALISTA in terms of test loss and convergence speed under high sparsity?
• RQ4 Does preserving the dictionary structure in SLISTA enable better generalization or scalability in structured settings?
• RQ5 Why does ALISTA fail to converge in the unsupervised setting, and how does this relate to the theoretical findings?

Key findings

• SLISTA outperforms LISTA and ALISTA in test loss for high regularization parameters (λ = 0.8), indicating superior performance in high-sparsity regimes.
• For low sparsity (low λ), LISTA performs best, as SLISTA cannot benefit from larger steps when solutions are less sparse.
• Theorem 4.4 confirms that the deepest layers of any converging LISTA network must reduce to ISTA with learned step sizes, validating the architectural choice of SLISTA.
• Empirical results show that in trained SLISTA networks, the ratio α^(t)/β^(t)W^(t) converges to D, confirming the theoretical alignment of weights with the dictionary.
• SLISTA preserves the dictionary structure, making it suitable for application in convolutional or wavelet-based sparse coding with fast matrix multiplications.
• ALISTA fails to converge in the unsupervised setting, consistent with Theorem 4.4, which shows that W_ALISTA cannot align with D unless step sizes are learned.