[Paper Review] Unrolled Optimization with Deep Priors

Steven Diamond, Vincent Sitzmann+2 more|arXiv (Cornell University)|May 22, 2017

Sparse and Compressive Sensing Techniques3 references128 citations

TL;DR

The paper introduces ODP, a principled framework that unrolls classical optimization with learnable deep CNN priors to solve inverse imaging problems, achieving state-of-the-art results in denoising, deblurring, and compressed sensing MRI.

ABSTRACT

A broad class of problems at the core of computational imaging, sensing, and low-level computer vision reduces to the inverse problem of extracting latent images that follow a prior distribution, from measurements taken under a known physical image formation model. Traditionally, hand-crafted priors along with iterative optimization methods have been used to solve such problems. In this paper we present unrolled optimization with deep priors, a principled framework for infusing knowledge of the image formation into deep networks that solve inverse problems in imaging, inspired by classical iterative methods. We show that instances of the framework outperform the state-of-the-art by a substantial margin for a wide variety of imaging problems, such as denoising, deblurring, and compressed sensing magnetic resonance imaging (MRI). Moreover, we conduct experiments that explain how the framework is best used and why it outperforms previous methods.

Motivation & Objective

• Motivate and formalize a general framework (ODP) that embeds priors into unrolled optimization for inverse imaging problems.
• Combine data consistency with a learned CNN prior to achieve high-quality reconstructions.
• Demonstrate state-of-the-art performance across denoising, deblurring, and compressed sensing MRI.
• Provide empirical guidance on when and how to exploit prior information in ODP.
• Offer insights into the choice of optimization algorithms and network architectures within the unrolled framework.

Proposed method

• Unrolled an optimization algorithm (e.g., proximal gradient) into a fixed-depth network with alternating data-step and CNN-prior steps.
• Parameterize the proximal operator or gradient of the prior term via a CNN prior (often a residual CNN).
• Train the network end-to-end using a reconstruction loss over image–measurement pairs, optimizing over CNN and algorithm hyperparameters.
• Initialize with a simple backprojection x^0 = A^H y to solve inverse problems.
• Compare different unrolled optimization variants (proximal gradient, ADMM, LADMM, gradient descent) to assess which best leverages the data term.
• Provide design defaults: choose proximal-gradient unroll as a strong baseline, use residual CNN priors, and adopt decaying step-size heuristics inspired by HQS.

Experimental results

Research questions

• RQ1 Can unrolled optimization with learned priors outperform state-of-the-art methods across common imaging inverse problems (denoising, deblurring, CS-MRI)?
• RQ2 How should the data-term inversion be integrated into the unrolled network (which optimization algorithm and what proximal/gradient steps work best)?
• RQ3 To what extent does the CNN prior contribute to performance versus the data-step, and how do these contributions interact across different problems?
• RQ4 Do architectures and parameter choices learned within ODP transfer across image formation operators (i.e., generalize beyond a single A)?

Key findings

• ODP models outperform state-of-the-art methods on denoising, deblurring, and CS-MRI across multiple experiments.
• A single ODP model trained across multiple image formation operators can outperform specialized models trained per operator (e.g., in CS-MRI).
• Proximal-gradient-based unrolled networks with CNN priors and operators that approximately invert A per iteration tend to perform best among tested algorithms.
• The data-step’s inversion, when feasible and accurate, significantly aids reconstruction; CNN priors mainly denoise and correct residual artifacts.
• Prior information generally improves generalization, especially when the image-formation operator is complex or unseen during training.
• In ablations, pure CNN priors without a data step underperform for deblurring and CS-MRI, highlighting the importance of the data-consistency term in challenging inverse problems.