[Paper Review] Scalable Plug-and-Play ADMM with Convergence Guarantees

Yu Sun, Zihui Wu+3 more|arXiv (Cornell University)|Jun 5, 2020

Sparse and Compressive Sensing Techniques72 references4 citations

TL;DR

This paper proposes an incremental plug-and-play ADMM (IPA) algorithm that enables scalable, convergence-guaranteed solving of large-scale inverse problems by processing data in small batches. Unlike prior PnP methods, IPA maintains low memory and per-iteration cost independent of dataset size, supports nonsmooth data-fidelity terms, and achieves faster convergence than PnP-SGD while ensuring theoretical convergence under explicit assumptions.

ABSTRACT

Plug-and-play priors (PnP) is a broadly applicable methodology for solving inverse problems by exploiting statistical priors specified as denoisers. Recent work has reported the state-of-the-art performance of PnP algorithms using pre-trained deep neural nets as denoisers in a number of imaging applications. However, current PnP algorithms are impractical in large-scale settings due to their heavy computational and memory requirements. This work addresses this issue by proposing an incremental variant of the widely used PnP-ADMM algorithm, making it scalable to large-scale datasets. We theoretically analyze the convergence of the algorithm under a set of explicit assumptions, extending recent theoretical results in the area. Additionally, we show the effectiveness of our algorithm with nonsmooth data-fidelity terms and deep neural net priors, its fast convergence compared to existing PnP algorithms, and its scalability in terms of speed and memory.

Motivation & Objective

• To address the impractical computational and memory demands of existing plug-and-play PnP algorithms in large-scale settings.
• To develop a scalable variant of PnP-ADMM that processes data incrementally, reducing memory and per-iteration complexity independent of dataset size.
• To extend theoretical convergence guarantees to incremental processing of data, covering both smooth and nonsmooth data-fidelity terms.
• To enable the use of deep neural net priors in large-scale problems with faster convergence than existing stochastic alternatives like PnP-SGD.
• To provide a theoretically sound, flexible, and practical framework for large-scale image reconstruction and inverse problems.

Proposed method

• Proposes an incremental PnP-ADMM (IPA) algorithm that processes one or a small subset of data elements at a time, rather than the full dataset in batch mode.
• Uses a randomized, incremental update scheme where the data-fidelity term is processed in a cyclic or stochastic manner over data subsets.
• Maintains the ADMM structure with alternating minimization over primal and dual variables, but applies the proximal operator for the data-fidelity term only to a subset of measurements at each step.
• Employs a denoiser (e.g., pre-trained deep neural net) as the prior, applied in the same way as in standard PnP-ADMM, but adapted to incremental updates.
• Theoretical analysis relies on firm nonexpansiveness and fixed-point convergence of the underlying operator, extending prior results from batch to incremental processing.
• Establishes convergence under explicit assumptions: Lipschitz continuity of the data-fidelity gradient, boundedness of iterates, and firm nonexpansiveness of the denoiser and proximal operators.

Experimental results

Research questions

• RQ1 Can PnP-ADMM be made scalable to large datasets without sacrificing convergence guarantees?
• RQ2 Does incremental processing of data in PnP-ADMM preserve convergence when using nonsmooth data-fidelity terms and learned denoisers?
• RQ3 How does the convergence rate and memory usage of the incremental PnP-ADMM compare to batch PnP-ADMM and PnP-SGD?
• RQ4 Can the theoretical convergence framework for batch PnP-ADMM be extended to partial, randomized data processing?
• RQ5 What is the empirical performance of the incremental algorithm in terms of speed, memory efficiency, and reconstruction quality on large-scale imaging problems?

Key findings

• The proposed IPA algorithm achieves convergence to a fixed point under explicit assumptions, extending the theoretical guarantees of batch PnP-ADMM to incremental processing.
• IPA maintains convergence even with nonsmooth data-fidelity terms, such as the ℓ₁-norm, which are not supported by PnP-SGD in the same theoretical framework.
• IPA demonstrates faster convergence than PnP-SGD in numerical experiments, particularly in early iterations, due to the use of ADMM's alternating direction structure.
• The memory and per-iteration computational cost of IPA are independent of the total number of measurements, enabling scalability to very large datasets.
• Numerical validation confirms that IPA effectively integrates deep neural net priors with nonsmooth data-fidelity terms, achieving state-of-the-art performance in large-scale image reconstruction tasks.
• Theoretical analysis shows that the incremental operator remains firmly nonexpansive under the same assumptions as batch PnP-ADMM, ensuring convergence via Krasnosel’skii-Mann theorem.