[Paper Review] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps

Cheng Lü, Yuhao Zhou+4 more|arXiv (Cornell University)|Jun 2, 2022

Model Reduction and Neural NetworksPhysics and Astronomy288 citations

TL;DR

DPM-Solver introduces a fast, training-free solver for diffusion probabilistic models by solving diffusion ODEs with high-order exponential integrators, enabling high-quality samples in about 10 steps. It outperforms prior training-free samplers across datasets.

ABSTRACT

Diffusion probabilistic models (DPMs) are emerging powerful generative models. Despite their high-quality generation performance, DPMs still suffer from their slow sampling as they generally need hundreds or thousands of sequential function evaluations (steps) of large neural networks to draw a sample. Sampling from DPMs can be viewed alternatively as solving the corresponding diffusion ordinary differential equations (ODEs). In this work, we propose an exact formulation of the solution of diffusion ODEs. The formulation analytically computes the linear part of the solution, rather than leaving all terms to black-box ODE solvers as adopted in previous works. By applying change-of-variable, the solution can be equivalently simplified to an exponentially weighted integral of the neural network. Based on our formulation, we propose DPM-Solver, a fast dedicated high-order solver for diffusion ODEs with the convergence order guarantee. DPM-Solver is suitable for both discrete-time and continuous-time DPMs without any further training. Experimental results show that DPM-Solver can generate high-quality samples in only 10 to 20 function evaluations on various datasets. We achieve 4.70 FID in 10 function evaluations and 2.87 FID in 20 function evaluations on the CIFAR10 dataset, and a $4\sim 16 imes$ speedup compared with previous state-of-the-art training-free samplers on various datasets.

Motivation & Objective

• Motivate faster sampling for diffusion probabilistic models (DPMs) without extra training.
• Leverage the diffusion ODE perspective to exploit semi-linear structure for exact linear term handling.
• Develop high-order, few-step solvers with convergence guarantees for DPMs.
• Provide adaptive and discrete-time compatibility to cover continuous and discrete DPMs.

Proposed method

• Formulate diffusion sampling as solving a diffusion ODE with semi-linear structure.
• Derive exact solution for the linear part via variation of constants and transform to an exponentially weighted integral of the noise predictor.
• Introduce DPM-Solver with first-, second-, and third-order versions (DPM-Solver-1/2/3) and convergence guarantees.
• Use an adaptive or uniform step-size strategy and combine solvers to achieve few-step sampling (NFE ~ 10-20).
• Show equivalence of DPM-Solver-1 with DDIM updates, and compare against RK-based solvers and training-based methods.

Experimental results

Research questions

• RQ1 Can diffusion probabilistic model sampling be cast as a diffusion ODE with a semi-linear structure to enable exact handling of the linear term?
• RQ2 What high-order, training-free solvers can achieve quality samples with around 10 steps across datasets?
• RQ3 Do exponential-integrator-inspired solvers provide convergence guarantees for DPMs in few-step regimes?
• RQ4 Is there a practical step-size schedule (adaptive/uniform) that preserves sample quality while minimizing NFEs?
• RQ5 Can the approach extend to continuous-time and discrete-time DPMs, including classifier-guided sampling?

Key findings

• DPM-Solver achieves high-quality samples with about 10 to 20 function evaluations (NFE) across datasets.
• DPM-Solver-1, -2, and -3 provide first-, second-, and third-order convergence guarantees for diffusion ODEs.
• DPM-Solver outperforms prior training-free samplers and traditional RK-based methods in few-step regimes, e.g., CIFAR-10 results show faster sample quality gains.
• DDIM is shown to be a special case of DPM-Solver-1, explaining its performance through semi-linear ODE structure.
• Adaptive step-size strategies and solver combinations maximize efficiency under fixed NFE budgets.