[Paper Review] Denoising Diffusion Implicit Models

Jiaming Song, Chenlin Meng+1 more|arXiv (Cornell University)|Oct 6, 2020

Generative Adversarial Networks and Image SynthesisComputer Science40 references102 citations

TL;DR

DDIMs are implicit diffusion models that reuse the DDPM training objective but enable non-Markovian forward processes, allowing high-quality image generation with far fewer sampling steps and enabling latent-space interpolation and reconstruction.

ABSTRACT

Denoising diffusion probabilistic models (DDPMs) have achieved high quality image generation without adversarial training, yet they require simulating a Markov chain for many steps to produce a sample. To accelerate sampling, we present denoising diffusion implicit models (DDIMs), a more efficient class of iterative implicit probabilistic models with the same training procedure as DDPMs. In DDPMs, the generative process is defined as the reverse of a Markovian diffusion process. We construct a class of non-Markovian diffusion processes that lead to the same training objective, but whose reverse process can be much faster to sample from. We empirically demonstrate that DDIMs can produce high quality samples $10 imes$ to $50 imes$ faster in terms of wall-clock time compared to DDPMs, allow us to trade off computation for sample quality, and can perform semantically meaningful image interpolation directly in the latent space.

Motivation & Objective

• Motivate the gap between DDPM sampling efficiency and GAN-like performance.
• Introduce a broader class of forward processes that preserve the DDPM training objective.
• Propose DDIMs as deterministic implicit models enabling faster, controllable generation.
• Show empirical benefits including faster sampling, interpolation, and reconstruction capabilities.

Proposed method

• Generalize the forward diffusion from Markovian to non-Markovian while preserving the same surrogate objective as DDPM.
• Define a trainable generative process p_theta(x_0:T) that uses predictions of x_0 from x_t via f_theta^(t)(x_t).
• Derive a unified variational objective J_sigma that is equivalent to the DDPM L_gamma objective for suitable gamma, enabling reuse of DDPM training across non-Markovian forwards.
• Derive the DDIM sampling update that combines predicted x_0 with a deterministic path towards x_{t-1}, enabling deterministic (eta=0) or stochastic sampling (eta>0).
• Propose accelerated generation by using forward-subset trajectories tau, to sample with fewer steps without retraining.

Experimental results

Research questions

• RQ1 Can non-Markovian forward processes yield the same variational objective as DDPMs while enabling faster or more controllable sampling?
• RQ2 How can we construct a deterministic or semi-deterministic sampling scheme that preserves high-level semantics and enables latent-space interpolation?
• RQ3 What are the trade-offs between sampling speed, sample quality, and reconstruction capabilities in DDIMs compared to DDPMs?
• RQ4 Can pretrained DDPM-trained models be repurposed for non-Markovian forward processes without retraining?
• RQ5 Is there a continuous-time ODE interpretation of the DDIM sampler that connects to neural ODEs and probability flow methods?

Key findings

• DDIMs achieve 10x to 50x faster sampling than DDPMs while maintaining high sample quality.
• DDIMs exhibit a consistency property where same initial latent x_T yields samples with similar high-level features across varying trajectory lengths.
• Deterministic DDIM sampling enables meaningful latent-space interpolations between samples.
• DDIMs can reconstruct observations from latent codes with low error, unlike stochastic DDPM sampling.
• Faster generation with reduced steps does not require retraining, since DDIMs reuse the DDPM objective with a non-Markovian forward process.
• Interpolation and reconstruction capabilities resemble implicit models and neural ODE-like behavior.