[Paper Review] Accelerated Forward-Backward Optimization using Deep Learning

Sebastian Banert, Jevgenija Rudzusika+2 more|arXiv (Cornell University)|May 11, 2021

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper proposes a deep learning-enhanced forward-backward optimization framework that guarantees convergence by constraining learned updates to specific sets, ensuring superior performance over FISTA and ISTA—especially in early iterations—while maintaining theoretical convergence. The method uses a neural network to learn optimal updates within provably convergent parameter sets.

ABSTRACT

We propose several deep-learning accelerated optimization solvers with convergence guarantees. We use ideas from the analysis of accelerated forward-backward schemes like FISTA, but instead of the classical approach of proving convergence for a choice of parameters, such as a step-size, we show convergence whenever the update is chosen in a specific set. Rather than picking a point in this set using some predefined method, we train a deep neural network to pick the best update. Finally, we show that the method is applicable to several cases of smooth and non-smooth optimization and show superior results to established accelerated solvers.

Motivation & Objective

• Address the challenge of accelerating large-scale non-smooth convex optimization in scientific computing and inverse problems.
• Overcome the lack of convergence guarantees in existing learned optimization methods like LISTA.
• Develop a framework that combines deep learning with provable convergence for forward-backward splitting schemes.
• Enable faster convergence in early iterations while ensuring global convergence through constrained update learning.
• Extend the applicability of learned optimization to non-smooth problems such as sparsity-promoting regularization in computed tomography (CT).

Proposed method

• Formulate optimization as minimizing a sum of a smooth, differentiable convex function and a non-smooth, proper, convex, lower semi-continuous function.
• Use a deviation-based approach: instead of learning fixed parameters, learn an entire update function within a set of full dimension that guarantees convergence.
• Define a set of valid updates based on convergence theory of accelerated forward-backward schemes (e.g., FISTA), ensuring convergence whenever updates lie within this set.
• Train a deep neural network to select the optimal update from this valid set, using supervised learning with ground-truth solutions.
• Introduce a normalization constraint (eq. 37) to control the magnitude of learned deviations, ensuring convergence while improving performance.
• Apply the method to non-smooth optimization problems, particularly sparsity-promoting regularization in CT image reconstruction, using a learned forward-backward scheme.

Experimental results

Research questions

• RQ1 Can deep learning be used to accelerate forward-backward optimization while preserving theoretical convergence guarantees?
• RQ2 How can the space of valid updates be defined such that convergence is guaranteed regardless of the specific choice within that set?
• RQ3 Can a neural network effectively learn optimal updates from within a high-dimensional, convergent update set to outperform classical solvers like FISTA?
• RQ4 What is the impact of normalization parameters (e.g., α) on convergence speed and early-iteration performance?
• RQ5 Can the proposed method achieve faster convergence than FISTA and ISTA in practical applications like CT image reconstruction?

Key findings

• The proposed method outperforms both ISTA and FISTA in the first 10 iterations of optimization, especially in early convergence speed.
• The learned scheme with α = 0.5 achieved the best performance among tested normalization parameters in non-smooth optimization, balancing speed and convergence.
• Normalization via equation 37 significantly improves performance after 10 iterations and ensures global convergence.
• The method converges to the exact solution, confirming theoretical convergence guarantees despite being learned via deep neural networks.
• The approach generalizes well to non-smooth problems such as sparsity-promoting regularization in computed tomography (CT) reconstruction.
• The framework enables a significant performance gain over classical solvers without sacrificing convergence, even when only a few iterations are used.