[Paper Review] Fast Sampling of Diffusion Models with Exponential Integrator

Qinsheng Zhang, Yongxin Chen|arXiv (Cornell University)|Apr 29, 2022

Model Reduction and Neural Networks58 citations

TL;DR

DEIS introduces a diffusion exponential integrator sampler to accelerate diffusion model sampling, achieving high fidelity with as few as 10 NFEs and improving over DDIM across various settings.

ABSTRACT

The past few years have witnessed the great success of Diffusion models~(DMs) in generating high-fidelity samples in generative modeling tasks. A major limitation of the DM is its notoriously slow sampling procedure which normally requires hundreds to thousands of time discretization steps of the learned diffusion process to reach the desired accuracy. Our goal is to develop a fast sampling method for DMs with a much less number of steps while retaining high sample quality. To this end, we systematically analyze the sampling procedure in DMs and identify key factors that affect the sample quality, among which the method of discretization is most crucial. By carefully examining the learned diffusion process, we propose Diffusion Exponential Integrator Sampler~(DEIS). It is based on the Exponential Integrator designed for discretizing ordinary differential equations (ODEs) and leverages a semilinear structure of the learned diffusion process to reduce the discretization error. The proposed method can be applied to any DMs and can generate high-fidelity samples in as few as 10 steps. In our experiments, it takes about 3 minutes on one A6000 GPU to generate $50k$ images from CIFAR10. Moreover, by directly using pre-trained DMs, we achieve the state-of-art sampling performance when the number of score function evaluation~(NFE) is limited, e.g., 4.17 FID with 10 NFEs, 3.37 FID, and 9.74 IS with only 15 NFEs on CIFAR10. Code is available at https://github.com/qsh-zh/deis

Motivation & Objective

• Motivate fast sampling for diffusion models and analyze sources of sampling error.
• Identify discretization schemes that minimize error in the backward diffusion integration.
• Propose and develop the Diffusion Exponential Integrator Sampler (DEIS) with variants.
• Connect DEIS to existing samplers and provide theoretical justifications (e.g., DDIM as a special case).
• Validate DEIS empirically on standard datasets showing improved efficiency and fidelity.

Proposed method

• Analyze marginal-equivalent SDEs/ODEs for diffusion models and identify discretization errors.
• Introduce Exponential Integrator (EI) for semilinear ODEs in the probability flow and apply it to DEIS.
• Parameterize the score using epsilon_theta and L_t to stabilize learning and sampling.
• Develop DEIS variants including t AB-DEIS, rho RK-DEIS, and rho AB-DEIS via ODE transformations.
• Incorporate high-order polynomial extrapolation of epsilon_theta to improve accuracy (P_r extrapolation).
• Show that deterministic DDIM is a special case of DEIS under VPSDE, providing a discretization-based justification.

Experimental results

Research questions

• RQ1 How do fitting and discretization errors influence diffusion model sampling quality?
• RQ2 Can an EI-based discretization reduce discretization error and enable high-quality samples with few NFEs?
• RQ3 Is DDIM equivalent to a DEIS discretization under certain forward processes, and can DEIS generalize beyond VPSDE?
• RQ4 Do higher-order polynomial extrapolations of epsilon_theta further improve sampling accuracy with limited NFEs?

Key findings

• DEIS can produce high-fidelity samples with a small number of NFEs (e.g., 4.17 FID with 10 NFEs and 2.86 FID with 20 NFEs on CIFAR-10).
• EI-based discretization outperforms Euler and standard solvers by better handling the semilinear structure and score approximation error.
• Parameterizing the score as s_theta = -L_t^{-T} epsilon_theta and using epsilon_theta improves accuracy over direct score targets.
• Deterministic DDIM emerges as a special case of DEIS under VPSDE, offering a principled discretization perspective on its effectiveness.
• High-order polynomial extrapolation (P_r) of epsilon_theta further reduces discretization error and improves sampling with few steps.
• DEIS variants (rho RK-DEIS, rho AB-DEIS, t AB-DEIS) consistently outperform traditional samplers (DDPM, DDIM, PNDM, FastDPM) under limited NFEs.