[Paper Review] Denoising Diffusion Restoration Models

Bahjat Kawar, Michael Elad+2 more|arXiv (Cornell University)|Jan 27, 2022

Image and Signal Denoising Methods240 citations

TL;DR

DDRM is an unsupervised, diffusion-based solver for general linear inverse problems that delivers high-quality image restoration with few iterations, using pre-trained diffusion models and SVD-based problem conditioning.

ABSTRACT

Many interesting tasks in image restoration can be cast as linear inverse problems. A recent family of approaches for solving these problems uses stochastic algorithms that sample from the posterior distribution of natural images given the measurements. However, efficient solutions often require problem-specific supervised training to model the posterior, whereas unsupervised methods that are not problem-specific typically rely on inefficient iterative methods. This work addresses these issues by introducing Denoising Diffusion Restoration Models (DDRM), an efficient, unsupervised posterior sampling method. Motivated by variational inference, DDRM takes advantage of a pre-trained denoising diffusion generative model for solving any linear inverse problem. We demonstrate DDRM's versatility on several image datasets for super-resolution, deblurring, inpainting, and colorization under various amounts of measurement noise. DDRM outperforms the current leading unsupervised methods on the diverse ImageNet dataset in reconstruction quality, perceptual quality, and runtime, being 5x faster than the nearest competitor. DDRM also generalizes well for natural images out of the distribution of the observed ImageNet training set.

Motivation & Objective

• Motivate unsupervised restoration for diverse degradation models without retraining.
• Leverage pre-trained diffusion models as priors to sample from the posterior of inverse problems.
• Develop a variational-inference framework that aligns diffusion denoising with linear inverse problem constraints.
• Propose spectral-space diffusion conditioned on degradation via SVD to handle various linear operators.
• Demonstrate efficiency and robustness across tasks and datasets, including out-of-distribution images.

Proposed method

• Formulate a posterior sampling objective for linear inverse problems conditioned on measurements.
• Perform diffusion in the spectral space of the degradation operator via SVD to separate observed/missing components.
• Define p_theta and q_t as Gaussian conditionals matching a DDPM/DDIM-like objective under DDRM, enabling reuse of unconditional diffusion models.
• Prove equivalence of the DDRM objective to DDPM/DDIM under reasonable assumptions, allowing a single diffusion model to address multiple problems.
• Introduce efficiency improvements: accelerated sampling with subset timesteps, memory-efficient SVD, and compatibility with pre-trained diffusion models.
• Provide practical algorithmic steps (update rules) for solving restoration tasks with 20-100 NFEs.

Experimental results

Research questions

• RQ1 Can DDRM solve a wide range of linear inverse problems unsupervised, without re-training for each degradation model?
• RQ2 How can diffusion models be leveraged in the spectral domain of the degradation operator to faithfully reconstruct missing data while respecting measurements?
• RQ3 What are the performance, speed, and memory characteristics of DDRM compared to other unsupervised priors on diverse datasets?
• RQ4 Is a single diffusion model capable of handling multiple restoration tasks (super-resolution, deblurring, inpainting, colorization) including noisy measurements?

Key findings

• DDRM outperforms leading unsupervised priors on ImageNet across multiple tasks and often with far fewer NFEs (e.g., 20 vs 1500).
• DDRMs can achieve competitive PSNR/SSIM and much lower KID than baselines in noiseless and noisy settings.
• DDRM demonstrates strong qualitative results, producing diverse valid restorations and generalizing to out-of-distribution natural images.
• Using SVD, the method handles various degradation operators efficiently with memory usage optimized to O(n) in many cases.
• Theoretical result: with appropriate scheduling, the DDRM objective aligns with DDPM/DDIM objectives, enabling reuse of unconditional diffusion models for conditioned restoration.
• DDRMs maintain performance across different datasets and degradation models, demonstrating flexibility without task-specific retraining.