[Paper Review] Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise

Arpit Bansal, Eitan Borgnia+7 more|arXiv (Cornell University)|Aug 19, 2022

Generative Adversarial Networks and Image SynthesisComputer Science105 citations

TL;DR

The paper shows that diffusion-like generative modeling can be built around arbitrary deterministic image degradations (not just Gaussian noise) and introduces a stable sampling method (Algorithm 2) to invert these degradations for high-quality image generation and restoration.

ABSTRACT

Standard diffusion models involve an image transform -- adding Gaussian noise -- and an image restoration operator that inverts this degradation. We observe that the generative behavior of diffusion models is not strongly dependent on the choice of image degradation, and in fact an entire family of generative models can be constructed by varying this choice. Even when using completely deterministic degradations (e.g., blur, masking, and more), the training and test-time update rules that underlie diffusion models can be easily generalized to create generative models. The success of these fully deterministic models calls into question the community's understanding of diffusion models, which relies on noise in either gradient Langevin dynamics or variational inference, and paves the way for generalized diffusion models that invert arbitrary processes. Our code is available at https://github.com/arpitbansal297/Cold-Diffusion-Models

Motivation & Objective

• Explore generalized diffusion models built around arbitrary degradations beyond Gaussian noise.
• Train restoration networks to invert these degradations using a simple l1 loss.
• Develop robust sampling procedures that yield high-quality generations from deterministic degradations.

Proposed method

• Define a degradation operator D that transforms x0 into xt with severity t and a restoration network R to approximate x0 from xt.
• Train R by minimizing E_x ||R(D(x,t),t) - x|| with an l1 loss.
• Propose Algorithm 2 for sampling to counteract imperfect inversion, ensuring x_{s-1} = x_s - D(R(x_s,s),s) + D(R(x_s,s),s-1).
• Prove that for a class of linear degradations, Algorithm 2 recovers x_s = D(x0,s) even when R is not perfect.
• Demonstrate generation and inversion across deblurring, inpainting, super-resolution, snowification, and other transformations on MNIST, CIFAR-10, and CelebA.
• Evaluate with FID, SSIM, and RMSE to compare degraded, direct reconstruction, and sampled reconstruction.

Experimental results

Research questions

• RQ1 Can diffusion models be generalized to arbitrary deterministic degradations beyond Gaussian noise?
• RQ2 Does a stable sampling algorithm exist that enables high-quality generation when the restoration model is imperfect?
• RQ3 How do different deterministic transforms (deblurring, inpainting, super-resolution, snow) affect reconstruction quality and distributional similarity?
• RQ4 Can cold diffusion support unconditional generation using non-noise degradations like blur or masking?

Key findings

• A fully deterministic diffusion framework can generate photo-realistic images by alternating between restoration and degradation without using randomness.
• Algorithm 2 offers stability to restoration errors and can recover D(x0,s) for linear degradations even if R is imperfect.
• Deblurring, inpainting, and super-resolution models trained under this framework produce improved FID scores relative to direct reconstructions, indicating closer alignment to the data manifold.
• In snow and blur tasks, the sampled reconstructions can outperform direct reconstructions in distributional similarity (FID) metrics, despite sometimes increasing RMSE or lowering PSNR.
• The approach can extend to generation using blur, with results suggesting high fidelity but variable diversity, and can be extended to other transforms like animorphosis.