[Paper Review] Improved Denoising Diffusion Probabilistic Models

Alex Nichol, Prafulla Dhariwal|arXiv (Cornell University)|Feb 18, 2021

Generative Adversarial Networks and Image SynthesisComputer Science27 references412 citations

TL;DR

This paper enhances DDPMs to achieve competitive log-likelihoods, enables faster sampling with learned variances, and shows diffusion models cover more modes than GANs, scaling with compute.

ABSTRACT

Denoising diffusion probabilistic models (DDPM) are a class of generative models which have recently been shown to produce excellent samples. We show that with a few simple modifications, DDPMs can also achieve competitive log-likelihoods while maintaining high sample quality. Additionally, we find that learning variances of the reverse diffusion process allows sampling with an order of magnitude fewer forward passes with a negligible difference in sample quality, which is important for the practical deployment of these models. We additionally use precision and recall to compare how well DDPMs and GANs cover the target distribution. Finally, we show that the sample quality and likelihood of these models scale smoothly with model capacity and training compute, making them easily scalable. We release our code at https://github.com/openai/improved-diffusion

Motivation & Objective

• Motivate evaluating DDPMs on log-likelihood and distribution coverage.
• Improve DDPM log-likelihood while maintaining sample quality.
• Enable faster sampling by learning reverse-process variances.
• Investigate training objectives and noise schedules to reduce gradient noise.
• Show scalability of DDPMs with model size and compute.

Proposed method

• Introduce learned reverse-process variance via Sigma_theta as an interpolation between beta_t and tilde_beta_t (Equation 15).
• Propose a hybrid training objective L_hybrid = L_simple + lambda L_vlb to balance sample quality and likelihood.
• Replace the linear noise schedule with a cosine schedule to improve information retention during diffusion (Equation 17).
• Apply importance sampling to estimate L_vlb with reduced gradient noise (Equation 18).
• Compare training objectives (L_simple, L_hybrid, L_vlb) and schedules across ImageNet 64x64 and CIFAR-10 with ablations.
• Demonstrate faster sampling by enabling high-quality samples with fewer diffusion steps, using learned sigmas.

Experimental results

Research questions

• RQ1 Can DDPMs achieve competitive log-likelihoods on high-diversity datasets like ImageNet 64x64?
• RQ2 Does learning the reverse variances improve both likelihood and sample quality?
• RQ3 Does a cosine noise schedule improve information retention and sample quality over a linear schedule?
• RQ4 Can importance sampling reduce gradient noise in log-likelihood optimization?
• RQ5 How do DDPMs scale with model size and training compute in terms of FID and NLL?

Key findings

• Learned variances via Sigma_theta substantially improve log-likelihood while preserving sample quality.
• A hybrid objective with learned sigmas and cosine schedule yields better NLL and comparable FID to prior baselines.
• Importance sampling reduces gradient noise in L_vlb, enabling better log-likelihood optimization.
• Diffusion models achieve higher recall than GANs at similar FID, indicating broader mode coverage.
• Sampling speed improves: 100 steps can approach near-optimal FID for fully trained models.
• Model size and compute show predictable performance scaling for FID and NLL.