[Paper Review] Pseudo Numerical Methods for Diffusion Models on Manifolds

Luping Liu, Yi Ren+2 more|arXiv (Cornell University)|Feb 20, 2022

Advanced Neuroimaging Techniques and Applications125 citations

TL;DR

This paper reframes DDPMs as solving differential equations on manifolds and introduces pseudo numerical methods (PNDMs) that combine a gradient part from high-order methods with a nonlinear transfer part to accelerate sampling without sacrificing quality.

ABSTRACT

Denoising Diffusion Probabilistic Models (DDPMs) can generate high-quality samples such as image and audio samples. However, DDPMs require hundreds to thousands of iterations to produce final samples. Several prior works have successfully accelerated DDPMs through adjusting the variance schedule (e.g., Improved Denoising Diffusion Probabilistic Models) or the denoising equation (e.g., Denoising Diffusion Implicit Models (DDIMs)). However, these acceleration methods cannot maintain the quality of samples and even introduce new noise at a high speedup rate, which limit their practicability. To accelerate the inference process while keeping the sample quality, we provide a fresh perspective that DDPMs should be treated as solving differential equations on manifolds. Under such a perspective, we propose pseudo numerical methods for diffusion models (PNDMs). Specifically, we figure out how to solve differential equations on manifolds and show that DDIMs are simple cases of pseudo numerical methods. We change several classical numerical methods to corresponding pseudo numerical methods and find that the pseudo linear multi-step method is the best in most situations. According to our experiments, by directly using pre-trained models on Cifar10, CelebA and LSUN, PNDMs can generate higher quality synthetic images with only 50 steps compared with 1000-step DDIMs (20x speedup), significantly outperform DDIMs with 250 steps (by around 0.4 in FID) and have good generalization on different variance schedules. Our implementation is available at https://github.com/luping-liu/PNDM.

Motivation & Objective

• Motivate faster sampling for DDPMs without quality loss by treating diffusion as manifold-constrained differential equations.
• Develop pseudo numerical methods that use a nonlinear transfer part to stay on high-density data manifolds.
• Demonstrate that DDIMs are special cases of pseudo numerical methods and identify the best-performing variant.
• Show that PNDMs achieve large speedups (e.g., 20x) while maintaining or improving sample quality on standard datasets.

Proposed method

• Derive the ODE governing the DDPM reverse process under the manifold constraint.
• Split classical numerical methods into gradient (epsilon_theta) and transfer parts; introduce nonlinear transfer parts to form PN methods.
• Equate DDIMs to a simple PN method and propose pseudo linear multi-step methods (PLMS) and pseudo Runge-Kutta variants (PRK).
• Use linear multi-step as the main gradient part with a tailored transfer part to stay on the data manifold.
• Prove convergence: S/P-NDMs have third-order local error and are second-order convergent.
• Provide algorithmic descriptions for DDIMs, S-PNDMs, and F-PNDMs; discuss first three steps via Runge-Kutta before switching to PLMS.

Experimental results

Research questions

• RQ1 Can diffusion model sampling be accelerated by solving the reverse process on the data manifold using pseudo numerical methods?
• RQ2 Do pseudo numerical methods (PNDMs) outperform traditional numerical solvers and DDIMs in sample quality at reduced step counts?
• RQ3 How do the gradient and transfer components interact to maintain manifold-aligned samples and convergence guarantees?

Key findings

• PNDMs enable up to 20x faster sampling than 1000-step DDIMs on CIFAR-10 and CelebA without quality loss.
• PNDMs achieve a new state-of-the-art FID of 2.71 on CelebA with reduced steps.
• S-PNDMs and F-PNDMs improve FID notably at lower step counts across CIFAR-10 and CelebA and generalize across variance schedules.
• DDIMs are a special case of PN methods, providing a new interpretation of their performance.
• Theoretical analysis shows S/F-PNDMs have third-order local error and second-order convergence.