[Paper Review] Generative Modeling by Estimating Gradients of the Data Distribution

Yang Song, Stefano Ermon|arXiv (Cornell University)|Jul 12, 2019

Generative Adversarial Networks and Image Synthesis68 references985 citations

TL;DR

This paper learns a score function for perturbed data using score matching and generates samples via annealed Langevin dynamics, achieving competitive image generation results without adversarial training.

ABSTRACT

We introduce a new generative model where samples are produced via Langevin dynamics using gradients of the data distribution estimated with score matching. Because gradients can be ill-defined and hard to estimate when the data resides on low-dimensional manifolds, we perturb the data with different levels of Gaussian noise, and jointly estimate the corresponding scores, i.e., the vector fields of gradients of the perturbed data distribution for all noise levels. For sampling, we propose an annealed Langevin dynamics where we use gradients corresponding to gradually decreasing noise levels as the sampling process gets closer to the data manifold. Our framework allows flexible model architectures, requires no sampling during training or the use of adversarial methods, and provides a learning objective that can be used for principled model comparisons. Our models produce samples comparable to GANs on MNIST, CelebA and CIFAR-10 datasets, achieving a new state-of-the-art inception score of 8.87 on CIFAR-10. Additionally, we demonstrate that our models learn effective representations via image inpainting experiments.

Motivation & Objective

• Motivate a new generative modeling approach that avoids adversarial training and likelihood constraints.
• Address manifold and low-density sampling challenges by perturbing data with Gaussian noise across multiple levels.
• Learn a Noise Conditional Score Network that jointly estimates scores for all noise levels.
• Use annealed Langevin dynamics to sample from increasingly refined distributions and approach the data manifold.

Proposed method

• Estimate the score of perturbed data distributions via score matching without requiring normalized likelihoods.
• Train a single conditional score network s_theta(x, sigma) to approximate ∇x log q_sigma(x) for a set of Gaussian noise levels {sigma_i}.
• Combine denoising score matching across multiple noise levels with a weighted objective, using lambda(sigma_i) = sigma_i^2 to balance contributions.
• Sample with annealed Langevin dynamics that progressively reduces noise level, starting from large sigma to small sigma, to improve mixing and sample quality.
• Architect score networks using a U-Net with dilated convolutions and conditional instance normalization to handle image data.
• Provide a training objective that avoids adversarial training and can be used to compare different models quantitatively.

Experimental results

Research questions

• RQ1 Can score-based generative modeling learn data distributions without adversarial training or likelihood-based objectives?
• RQ2 Does perturbing data with Gaussian noise across multiple levels enable consistent score estimation and efficient sampling?
• RQ3 Can an annealed Langevin sampling procedure effectively generate high-quality samples from multi-noise score estimates?
• RQ4 Do Noise Conditional Score Networks (NCSNs) produce competitive image samples and useful representations (e.g., for inpainting) on standard datasets?

Key findings

• Achieved CIFAR-10 unconditional inception score of 8.87 (state-of-the-art at the time among unconditional models).
• Attained CIFAR-10 FID of 25.32, competitive with top models (e.g., SNGAN).
• Demonstrated high-quality samples on MNIST, CelebA, and CIFAR-10 that are comparable to likelihood-based models and GANs.
• Showed successful image inpainting demonstrating learned meaningful representations.
• Annealed Langevin dynamics with a multi-noise score network improves mode mixing over standard Langevin sampling.
• Provided a principled objective for model comparison without adversarial training.