[Paper Review] Interpreting and Improving Diffusion Models from an Optimization Perspective

Frank Permenter, Chenyang Yuan|arXiv (Cornell University)|Jun 8, 2023

Advanced Neuroimaging Techniques and Applications4 citations

TL;DR

This paper interprets denoising diffusion models as approximate gradient descent on the squared Euclidean distance function to the data manifold, enabling a deterministic optimization-based analysis. It proposes a new sampler that improves sample quality by aggregating denoiser outputs, achieving state-of-the-art FID scores on CIFAR-10 and CelebA with only 5–10 function evaluations.

ABSTRACT

Denoising is intuitively related to projection. Indeed, under the manifold hypothesis, adding random noise is approximately equivalent to orthogonal perturbation. Hence, learning to denoise is approximately learning to project. In this paper, we use this observation to interpret denoising diffusion models as approximate gradient descent applied to the Euclidean distance function. We then provide straight-forward convergence analysis of the DDIM sampler under simple assumptions on the projection error of the denoiser. Finally, we propose a new gradient-estimation sampler, generalizing DDIM using insights from our theoretical results. In as few as 5-10 function evaluations, our sampler achieves state-of-the-art FID scores on pretrained CIFAR-10 and CelebA models and can generate high quality samples on latent diffusion models.

Motivation & Objective

• To provide a deterministic, optimization-based interpretation of diffusion models by framing denoising as approximate projection onto the data manifold.
• To analyze the convergence of DDIM samplers under a relative-error model for the denoiser's projection accuracy.
• To develop a new sampling algorithm that improves sample quality by leveraging gradient estimation error control and denoiser output aggregation.
• To unify existing techniques like score distillation sampling and conditional generation under the framework of distance function optimization.
• To enable plug-in use of pre-trained denoisers in constrained optimization tasks via a principled distance minimization perspective.

Proposed method

• Interprets the denoiser output as an estimate of the gradient of the squared Euclidean distance function to the data manifold, i.e., ∇f(x) ≈ σεθ(x,σ).
• Introduces a relative-error model: ||∇f(x) − σεθ(x,σ)|| ≤ η||∇f(x)||, which bounds the error in gradient estimation.
• Shows that DDIM sampling is equivalent to approximate gradient descent on f(x) = ½ distK²(x), under the relative-error assumption.
• Derives conditions on the noise schedule σt that guarantee monotonic decrease of distK(xt) when error parameters (η,ν) are bounded.
• Proposes a new sampler (Algorithm 2) that aggregates multiple denoiser outputs to reduce gradient estimation error and improve convergence.
• Applies the framework to reinterpret and generalize techniques such as score distillation sampling and conditional generation as constrained optimization on the distance function.

Experimental results

Research questions

• RQ1 Can denoising in diffusion models be rigorously interpreted as approximate projection onto the data manifold?
• RQ2 How does the relative error in denoiser output affect the convergence of diffusion samplers?
• RQ3 Can the diffusion sampling process be reinterpreted as gradient descent on the squared distance function to the data manifold?
• RQ4 What noise schedule ensures convergence of the sampling process under bounded gradient estimation error?
• RQ5 Can improved samplers be designed by aggregating denoiser outputs to reduce estimation error?

Key findings

• The denoising process approximates projection onto the data manifold when the noise level σ is proportional to the distance to the manifold, under the manifold hypothesis.
• DDIM sampling is equivalent to approximate gradient descent on the squared Euclidean distance function, with convergence guaranteed under bounded relative error in gradient estimation.
• The proposed sampler achieves state-of-the-art FID scores on pretrained CIFAR-10 and CelebA models using only 5–10 function evaluations.
• The new sampler outperforms existing methods on high-resolution latent diffusion models such as Stable Diffusion, achieving competitive FID scores with minimal inference cost.
• The framework provides a new interpretation of score distillation sampling as gradient descent on a composite objective involving the distance function and a parameterized generator.
• The analysis reveals that training a denoiser is equivalent to learning a solution to the Eikonal equation ||∇d(x)|| = 1, linking diffusion to level-set and distance function learning.