[Paper Review] Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models

Fan Bao, Chongxuan Li+2 more|arXiv (Cornell University)|Jan 17, 2022

Generative Adversarial Networks and Image Synthesis74 citations

TL;DR

Analytic-DPM provides a training-free analytic estimate of the optimal reverse variance in diffusion probabilistic models, enabling faster and more likelihood-accurate sampling without additional training. It derives variance and KL bounds from the score function and uses dynamic programming to find efficient inference trajectories.

ABSTRACT

Diffusion probabilistic models (DPMs) represent a class of powerful generative models. Despite their success, the inference of DPMs is expensive since it generally needs to iterate over thousands of timesteps. A key problem in the inference is to estimate the variance in each timestep of the reverse process. In this work, we present a surprising result that both the optimal reverse variance and the corresponding optimal KL divergence of a DPM have analytic forms w.r.t. its score function. Building upon it, we propose Analytic-DPM, a training-free inference framework that estimates the analytic forms of the variance and KL divergence using the Monte Carlo method and a pretrained score-based model. Further, to correct the potential bias caused by the score-based model, we derive both lower and upper bounds of the optimal variance and clip the estimate for a better result. Empirically, our analytic-DPM improves the log-likelihood of various DPMs, produces high-quality samples, and meanwhile enjoys a 20x to 80x speed up.

Motivation & Objective

• Motivate faster, likelihood-consistent inference for diffusion probabilistic models (DPMs) by addressing reverse-variance estimation.
• Derive analytic forms for the optimal reverse variance and KL divergence in terms of the score function.
• Propose a training-free framework (Analytic-DPM) that estimates these quantities via Monte Carlo and a pretrained score model.
• Introduce bounds to clip variance estimates and reduce bias from score-model approximations.
• Demonstrate empirically that Analytic-DPM improves log-likelihood and sample quality while delivering substantial speedups.

Proposed method

• Prove that the optimal reverse mean and variance in the KL-minimizing reverse process have analytic forms dependent on the score function (Theorem 1).
• Estimate the optimal reverse variance 9 using Monte Carlo averages of the pretrained score model via a06Gamma and Eq. (9).
• Derive upper and lower bounds on the optimal variance (Theorem 2) and clip estimates to reduce bias.
• Construct a shorter forward trajectory and use a least-cost path DP (Watson et al., 2021) to determine an optimal sampling schedule (Eq. (14)).
• Extend the framework to continuous timesteps and discuss trajectory optimization (OT) and ET strategies.
• Validate the approach on pretrained and self-trained score models across datasets (CIFAR-10, CelebA, ImageNet) and compare against DDPM and DDIM baselines.

Experimental results

Research questions

• RQ1 Can the optimal reverse variance and KL divergence in a DPM be expressed analytically in terms of the score function?
• RQ2 Can a training-free Analytic-DPM estimate these optimal quantities efficiently from a pretrained score model?
• RQ3 How does clipping based on variance bounds affect bias and performance?
• RQ4 Does using a shortest or optimally-constructed trajectory (via DP) maintain or improve log-likelihood and sample quality with Analytic-DPM?
• RQ5 Is Analytic-DPM applicable across different forward processes (DDPM and DDIM) and datasets?

Key findings

• The optimal reverse mean and variance have analytic forms w.r.t. the score function (Theorem 1).
• The analytic variance estimate  is computable from a pretrained score model using Monte Carlo and Gamma statistics (Eq. (9)).
• Bounding and clipping the variance reduces bias, with explicit lower/upper bounds provided (Theorem 2).
• A shortest-path DP can efficiently select trajectories, enabling 20x to 80x speedups while improving or maintaining log-likelihood and sample quality across methods and datasets.
• Analytic-DPM consistently improves log-likelihood over original DDPM across CIFAR-10, CelebA-64, and ImageNet-64x64 under both ET and OT trajectories (Table 1); it also improves FID for DDIM-based sampling (Table 2).
• Analytic-DPM achieves substantial speedups (e.g., 40x on CIFAR-10 with ET/OT trajectories) while matching or surpassing baselines in likelihood and sample quality.