[Paper Review] Improved Techniques for Training Score-Based Generative Models

Yang Song, Stefano Ermon|arXiv (Cornell University)|Jun 16, 2020

Generative Adversarial Networks and Image Synthesis32 references142 citations

TL;DR

This paper analyzes scaling score-based generative models to high-resolution images and introduces techniques (noise scales, conditioning, EMA) enabling high-fidelity samples comparable to GANs on 64x64 to 256x256 images.

ABSTRACT

Score-based generative models can produce high quality image samples comparable to GANs, without requiring adversarial optimization. However, existing training procedures are limited to images of low resolution (typically below 32x32), and can be unstable under some settings. We provide a new theoretical analysis of learning and sampling from score models in high dimensional spaces, explaining existing failure modes and motivating new solutions that generalize across datasets. To enhance stability, we also propose to maintain an exponential moving average of model weights. With these improvements, we can effortlessly scale score-based generative models to images with unprecedented resolutions ranging from 64x64 to 256x256. Our score-based models can generate high-fidelity samples that rival best-in-class GANs on various image datasets, including CelebA, FFHQ, and multiple LSUN categories.

Motivation & Objective

• Explain limitations of prior score-based models on high-resolution images.
• Develop theoretically grounded methods to select noise scales and sampling parameters.
• Propose architecture and training tricks to improve stability and sample quality."
• Demonstrate scalability to 64×64–256×256 images across diverse datasets.

Proposed method

• Analytical guidance for choosing Gaussian noise scales from data distributions.
• Amortized score estimation by a single network across many noise scales (noise conditioning).
• Theoretical analysis of Langevin dynamics to optimize sampling performance across noise scales.
• Exponential moving average (EMA) of model parameters during sampling to improve stability.
• Denoising step (Tweede’s formula inspired) to improve final sample quality.
• Integration of the above into an end-to-end training and sampling recipe (NCSNv2).

Experimental results

Research questions

• RQ1 How can score-based models be scaled from 32×32 to high-resolution images (64×64–256×256)?
• RQ2 What noise scale configurations and conditioning methods yield reliable learning and fast, high-quality sampling?
• RQ3 Does exponential moving averaging of parameters stabilize training and improve sample fidelity?
• RQ4 Can a single amortized network handle many noise scales effectively?
• RQ5 What are the quantitative gains in standard metrics (FID/Inception) when applying these techniques across datasets?

Key findings

• NCSNv2 achieves high-fidelity samples on 64×64 CelebA and 128–256×256 LSUN/FFHQ datasets, surpassing prior score-based models.
• Optimal initial noise scale should be as large as the maximum pairwise distance in the training data to promote diversity.
• Geometric progression of noise scales with a specific ratio yields stable training and effective coverage of high-density regions.
• Rescaling the unconditional score network by 1/σ to incorporate noise information improves training with many scales.
• Choosing sampling steps and step size via a data-driven analysis reduces tuning and improves mixing.
• Exponential moving average of model parameters during sampling markedly stabilizes FID and reduces artifacts.