[Paper Review] Score-Based Generative Modeling through Stochastic Differential Equations

Yang Song, Jascha Sohl‐Dickstein+4 more|arXiv (Cornell University)|Nov 26, 2020

Generative Adversarial Networks and Image Synthesis48 references1,263 citations

TL;DR

Introduces a unified score-based generative modeling framework using forward and reverse stochastic differential equations (SDEs) to transform data to noise and back, enabling flexible sampling, exact likelihoods via neural ODEs, and state-of-the-art CIFAR-10 generation with continuous training and novel samplers.

ABSTRACT

Creating noise from data is easy; creating data from noise is generative modeling. We present a stochastic differential equation (SDE) that smoothly transforms a complex data distribution to a known prior distribution by slowly injecting noise, and a corresponding reverse-time SDE that transforms the prior distribution back into the data distribution by slowly removing the noise. Crucially, the reverse-time SDE depends only on the time-dependent gradient field (\aka, score) of the perturbed data distribution. By leveraging advances in score-based generative modeling, we can accurately estimate these scores with neural networks, and use numerical SDE solvers to generate samples. We show that this framework encapsulates previous approaches in score-based generative modeling and diffusion probabilistic modeling, allowing for new sampling procedures and new modeling capabilities. In particular, we introduce a predictor-corrector framework to correct errors in the evolution of the discretized reverse-time SDE. We also derive an equivalent neural ODE that samples from the same distribution as the SDE, but additionally enables exact likelihood computation, and improved sampling efficiency. In addition, we provide a new way to solve inverse problems with score-based models, as demonstrated with experiments on class-conditional generation, image inpainting, and colorization. Combined with multiple architectural improvements, we achieve record-breaking performance for unconditional image generation on CIFAR-10 with an Inception score of 9.89 and FID of 2.20, a competitive likelihood of 2.99 bits/dim, and demonstrate high fidelity generation of 1024 x 1024 images for the first time from a score-based generative model.

Motivation & Objective

• Motivate a unified diffusion-inspired framework to model data by continuously perturbing it with noise and reversing the process using score estimates.
• Develop methods to estimate time-dependent scores with neural networks and use SDEs to generate high-fidelity samples and compute exact likelihoods.
• Introduce sampling improvements (predictor-corrector, reverse diffusion samplers) and continuous training objectives to enhance sample quality and likelihood performance.
• Enable controllable generation for tasks like class-conditional generation, inpainting, and colorization within a single unconditional score model.
• Demonstrate scalability and results on CIFAR-10 and higher-resolution images, and compare with diffusion and score-based baselines.

Proposed method

• Model data with a continuous diffusion process (Itô SDE) that gradually adds noise from data distribution to a tractable prior.
• Derive the reverse-time SDE that uses the time-dependent score ∇x log pt(x) to transform noise back into data.
• Train a time-dependent score model sθ(x,t) via continuous score matching to approximate ∇x log pt(x).
• Solve the reverse SDE with general-purpose solvers; introduce Predictor-Corrector (PC) samplers combining numerical steps with score-based MCMC corrections.
• Derive and use a probability flow ODE that yields the same marginals as the SDE and enables exact likelihood computation via neural ODE techniques.
• Present and analyze Variance Exploding (VE), Variance Preserving (VP), and sub-VP SDE variants, including closed-form perturbation kernels and sampling updates.
• Demonstrate controllable generation by conditioning on auxiliary information (e.g., class labels) through forward-model gradients, enabling imputation and colorization.

Experimental results

Research questions

• RQ1 How can score-based generative modeling be unified under the framework of stochastic differential equations?
• RQ2 Can the reverse-time SDE be effectively estimated from time-dependent scores to generate high-fidelity samples?
• RQ3 What samplers (general SDE solvers, predictor-corrector, probability flow ODE) yield best trade-offs between sample quality, speed, and likelihood computation?
• RQ4 Can continuous training objectives and architectural improvements achieve state-of-the-art image generation metrics and exact likelihoods?
• RQ5 To what extent can unconditional score-based models support controllable generation tasks like class-conditional generation, inpainting, and colorization?

Key findings

• A unified SDE-based framework can map data to a noise prior and reverse the diffusion using estimated scores to generate data samples.
• Time-dependent score models trained via continuous score matching can approximate ∇x log pt(x) for all t, enabling reverse SDE sampling and exact likelihood via the probability flow ODE.
• Predictor-Corrector samplers and reverse diffusion samplers consistently improve sample quality over ancestral sampling across VE and VP/sub-VP SDEs.
• The probability flow ODE enables fast, adaptive sampling and exact likelihood computation, revealing improvements in log-likelihood (bits/dim) and competitive FID/IS on CIFAR-10.
• Architectural and training enhancements (NCSN++, DDPM++, continuous objective) achieve record image-generation metrics on CIFAR-10 (Inception score 9.89, FID 2.20) and enable 1024×1024 CelebA-HQ generation with score-based models.
• A new likelihood-record method (DDPM++ cont. with sub-VP) achieves 2.99 bits/dim on uniformly dequantized CIFAR-10, the best reported to date.