[Paper Review] Common Diffusion Noise Schedules and Sample Steps are Flawed

Shanchuan Lin, Bingchen Liu+2 more|arXiv (Cornell University)|May 15, 2023

Advanced Neuroimaging Techniques and Applications10 citations

TL;DR

The paper identifies flaws in common diffusion noise schedules and sampler start points, showing training/inference mismatch and brightness bias, and proposes fixes: zero terminal SNR schedules, v-prediction, starting from the last timestep, and classifier-free guidance rescaling.

ABSTRACT

We discover that common diffusion noise schedules do not enforce the last timestep to have zero signal-to-noise ratio (SNR), and some implementations of diffusion samplers do not start from the last timestep. Such designs are flawed and do not reflect the fact that the model is given pure Gaussian noise at inference, creating a discrepancy between training and inference. We show that the flawed design causes real problems in existing implementations. In Stable Diffusion, it severely limits the model to only generate images with medium brightness and prevents it from generating very bright and dark samples. We propose a few simple fixes: (1) rescale the noise schedule to enforce zero terminal SNR; (2) train the model with v prediction; (3) change the sampler to always start from the last timestep; (4) rescale classifier-free guidance to prevent over-exposure. These simple changes ensure the diffusion process is congruent between training and inference and allow the model to generate samples more faithful to the original data distribution.

Motivation & Objective

• Identify how common diffusion noise schedules fail to enforce zero terminal SNR and why this creates training/inference mismatch.
• Show how sampler implementations that don’t start from the last timestep exacerbate brightness bias.
• Propose practical fixes to align training and inference and improve sample quality.
• Validate fixes through training and evaluation on Stable Diffusion variants and standard datasets.

Proposed method

• Enforce zero terminal SNR by rescaling existing noise schedules in the sqrt(alpha_bar) space (Algorithm 1).
• Switch to v-prediction and v-loss when terminal SNR is zero to provide meaningful guidance (Equations 11–12).
• Require samplers to start from the last timestep to align training/inference (Table 2 reference and Section 3.3).
• Propose classifier-free guidance rescaling to prevent over-exposure (Equations 13–16, Algorithm 2).
• Train the model with the proposed schedule and sampling strategy and compare qualitatively and quantitatively to the baseline Stable Diffusion (Section 4).
• Discuss correct handling of v-prediction versus epsilon, avoiding epsilon-based formulations at zero SNR (Section 6).

(a) Flawed

Experimental results

Research questions

• RQ1 Do common diffusion noise schedules permit non-zero SNR at the final timestep, causing training/inference mismatch?
• RQ2 Does starting sampling from the last timestep improve alignment between training and inference and broaden the achievable brightness range?
• RQ3 Can simple schedule rescaling, v-prediction, and CFG rescaling fix the brightness and exposure issues observed in Stable Diffusion?
• RQ4 How do the proposed changes affect quantitative distributional alignment (FID/IS) and qualitative sample diversity?
• RQ5 What are practical considerations for sampler implementations when terminal SNR is zero?

Key findings

• Zero terminal SNR schedules remove leakage of low-frequency signal at the final step, aligning training and inference.
• V-prediction maintains meaningful loss guidance when terminal SNR is zero, with similar visual quality to epsilon prediction.
• Starting from the last timestep in samplers is crucial for consistency with training under zero-terminal-SNR schedules.
• Classifier-free guidance rescaling mitigates over-exposure as terminal SNR approaches zero (phi in [0.5,0.75] shown).
• Finetuned model with fixes yields improved FID/IS on COCO 2014 validation: 21.66 (FID) and 36.16 (IS) vs. 23.76 and 32.84 (official SD v2.1-base).
• Trailing sample-step selection is more efficient than Linspace when S is small; differences diminish with larger S.

(b) Corrected