[Paper Review] Denoising MCMC for Accelerating Diffusion-Based Generative Models

Beomsu Kim, Jong Chul Ye|arXiv (Cornell University)|Sep 29, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes Denoising MCMC (DMCMC), an orthogonal acceleration method for score-based generative models that uses Markov Chain Monte Carlo (MCMC) to sample from the product space of data and diffusion time, followed by reverse-S/ODE integration to denoise samples. By starting near the data manifold, DMCMC drastically reduces reverse-S/ODE integration cost, achieving state-of-the-art FID scores of 3.86 (≈10 NFE) on CIFAR10 and 6.99 (≈160 NFE) on CelebA-HQ-256.

ABSTRACT

Diffusion models are powerful generative models that simulate the reverse of diffusion processes using score functions to synthesize data from noise. The sampling process of diffusion models can be interpreted as solving the reverse stochastic differential equation (SDE) or the ordinary differential equation (ODE) of the diffusion process, which often requires up to thousands of discretization steps to generate a single image. This has sparked a great interest in developing efficient integration techniques for reverse-S/ODEs. Here, we propose an orthogonal approach to accelerating score-based sampling: Denoising MCMC (DMCMC). DMCMC first uses MCMC to produce samples in the product space of data and variance (or diffusion time). Then, a reverse-S/ODE integrator is used to denoise the MCMC samples. Since MCMC traverses close to the data manifold, the computation cost of producing a clean sample for DMCMC is much less than that of producing a clean sample from noise. To verify the proposed concept, we show that Denoising Langevin Gibbs (DLG), an instance of DMCMC, successfully accelerates all six reverse-S/ODE integrators considered in this work on the tasks of CIFAR10 and CelebA-HQ-256 image generation. Notably, combined with integrators of Karras et al. (2022) and pre-trained score models of Song et al. (2021b), DLG achieves SOTA results. In the limited number of score function evaluation (NFE) settings on CIFAR10, we have $3.86$ FID with $\approx 10$ NFE and $2.63$ FID with $\approx 20$ NFE. On CelebA-HQ-256, we have $6.99$ FID with $\approx 160$ NFE, which beats the current best record of Kim et al. (2022) among score-based models, $7.16$ FID with $4000$ NFE. Code: https://github.com/1202kbs/DMCMC

Motivation & Objective

• To accelerate score-based sampling in diffusion models by reducing the computational cost of reverse-S/ODE integration.
• To address the inefficiency of traditional MCMC methods in multi-modal, high-dimensional data distributions by enabling traversal across disconnected modes.
• To develop a plug-and-play framework that combines MCMC with reverse-S/ODE integrators for faster, higher-quality sample generation.
• To achieve state-of-the-art FID scores with minimal score function evaluations (NFE) on standard benchmarks like CIFAR10 and CelebA-HQ-256.

Proposed method

• Proposes Denoising MCMC (DMCMC), a framework that samples from the product space of data and diffusion time (variance) using MCMC to explore high-density regions near the data manifold.
• Uses a reverse-S/ODE integrator to denoise MCMC samples starting from a time close to zero (low noise), significantly shortening the integration interval compared to starting from maximum noise.
• Introduces Denoising Langevin Gibbs (DLG), a practical instance of DMCMC that alternates between Langevin dynamics for data updates and Gibbs sampling for noise level prediction.
• Employs a pre-trained noise-conditional score network and a lightweight noise level classifier, enabling efficient and scalable sampling with negligible additional computation.
• Treats the diffusion process as a time-dependent score function, allowing integration of reverse-S/ODEs from any initial time t_n near zero.
• Optimizes hyperparameters such as step size η, total NFE, and fraction of NFE allocated to denoising (n_den/n) to balance sample quality and diversity.

Experimental results

Research questions

• RQ1 Can MCMC sampling in the product space of data and diffusion time enable faster and more efficient reverse-S/ODE integration in score-based generative models?
• RQ2 Does starting reverse-S/ODE integration from a low-noise state, enabled by MCMC, significantly reduce the number of required score function evaluations (NFE) while maintaining or improving sample quality?
• RQ3 Can a plug-and-play framework combining MCMC and reverse-S/ODE integrators achieve state-of-the-art performance across diverse generative modeling benchmarks?
• RQ4 How do hyperparameters such as Langevin step size (η), total NFE, and denoising fraction (n_den/n) affect the trade-off between sample quality and diversity in the DMCMC framework?

Key findings

• DLG achieves a state-of-the-art FID of 3.86 on CIFAR10 with only approximately 10 NFE, outperforming prior methods that achieved 4.17 FID under the same NFE budget.
• On CelebA-HQ-256, DLG achieves a SOTA FID of 6.99 with approximately 160 NFE, surpassing the previous best result of 7.16 FID with 4,000 NFE using a different score-based model.
• The method reduces the average reverse-S/ODE integration interval length by starting from a noise level close to zero, enabling faster convergence with minimal additional computational cost.
• Ablation studies show that optimal performance is achieved when the denoising fraction (n_den/n) is intermediate (not near 0 or 1), and that a dimension-independent displacement parameter (κ = η/√d) enables effective hyperparameter transfer across datasets.
• The computation cost of the noise level classifier is negligible, making the acceleration effect effectively free in terms of additional FLOPs.
• The framework is general and plug-and-play: any MCMC, VE process, and reverse-S/ODE integrator can be used, demonstrating broad applicability and scalability.