[Paper Review] Plug-and-Play Methods for Integrating Physical and Learned Models in Computational Imaging

Ulugbek S. Kamilov, Charles A. Bouman+2 more|arXiv (Cornell University)|Mar 31, 2022

Mathematical Biology Tumor Growth24 citations

TL;DR

PnP methods integrate physical measurement models with learned priors by replacing proximal steps with denoisers, enabling flexible, modular reconstructions across imaging modalities. The paper reviews foundations, variants, convergence, and applications, and discusses equilibrium formulations and online/DU/DEQ extensions.

ABSTRACT

Plug-and-Play Priors (PnP) is one of the most widely-used frameworks for solving computational imaging problems through the integration of physical models and learned models. PnP leverages high-fidelity physical sensor models and powerful machine learning methods for prior modeling of data to provide state-of-the-art reconstruction algorithms. PnP algorithms alternate between minimizing a data-fidelity term to promote data consistency and imposing a learned regularizer in the form of an image denoiser. Recent highly-successful applications of PnP algorithms include bio-microscopy, computerized tomography, magnetic resonance imaging, and joint ptycho-tomography. This article presents a unified and principled review of PnP by tracing its roots, describing its major variations, summarizing main results, and discussing applications in computational imaging. We also point the way towards further developments by discussing recent results on equilibrium equations that formulate the problem associated with PnP algorithms.

Motivation & Objective

• Motivate and formalize the Plug-and-Play (PnP) framework as a modular approach to combining data fidelity with learned priors in inverse imaging problems.
• Summarize historical development, core algorithms (PnP-ADMM, PnP-FISTA), and extensions (online PnP, RED) with theoretical convergence insights.
• Highlight practical implementations, including turning a denoiser into a super-resolver and adapting priors across measurement operators.
• Discuss equilibrium formulations and connections to deep unfolding (DU) and deep equilibrium (DEQ) models.
• Identify limitations, trade-offs, and directions for future work in PnP methods.

Proposed method

• Describe the composite objective f(x)=g(x)+h(x) and proximal algorithms (ADMM, FISTA) as the basis for PnP variants.
• Replace the proximal operator prox_{γh} with a black-box denoiser D in ADMM/FISTA to create PnP-ADMM and PnP-FISTA.
• Explain how a pre-trained denoiser can act as an image prior and how data-fidelity updates separate from learned priors for modularity.
• Discuss convergence via consensus equilibrium and conditions (e.g., contractive operators, Lipschitz denoisers) for PnP/RED variants.
• Present online/mini-batch variants (Online PnP, SIMBA) to handle large-scale data with reduced per-iteration cost.
• Link DU/DEQ frameworks by unrolling or implicitly differentiating PnP iterations with learned priors (AR operators) trained for specific measurement models.

Experimental results

Research questions

• RQ1 How can learned denoisers be integrated with physical forward models to solve inverse imaging problems in a modular, plug-and-play fashion?
• RQ2 What are the convergence characteristics and theoretical guarantees for PnP methods under various denoiser and data-fidelity assumptions?
• RQ3 How can PnP be adapted to online/large-scale settings and to different measurement operators without retraining priors?
• RQ4 What is the role of equilibrium and DU/DEQ interpretations in understanding and improving PnP reconstructions?
• RQ5 What are practical performance gains when using problem-specific artifact-removal priors versus generic AWGN denoisers?

Key findings

• PnP replaces proximal steps with a denoiser, enabling modular integration of data fidelity and learned priors across imaging problems.
• PnP-ADMM and PnP-FISTA separate forward-model updates from denoising, enabling reuse of the same denoiser across different measurement operators.
• Equilibrium formulations (consensus equilibrium) provide a framework to analyze convergence when using black-box denoisers.
• Online PnP and SIMBA reduce per-iteration cost for large-scale problems by using minibatches or blocks of measurements, with convergence analyses under stochastic gradient assumptions.
• DU/DEQ extensions train denoisers in the context of the forward model, using either artifact-removal priors or end-to-end training, improving performance at the cost of reduced generality.
• Empirical results (e.g., Fig. 6) show that problem-specific artifact-removal priors can outperform generic AWGN denoisers in certain compressive sensing and IDT tomography scenarios, with relative PSNR/SSIM reported in figures.