[Paper Review] On the Importance of Noise Scheduling for Diffusion Models

Ting Chen|arXiv (Cornell University)|Jan 26, 2023

Advanced Neuroimaging Techniques and Applications33 citations

TL;DR

The paper empirically analyzes noise scheduling strategies for diffusion models, proposes a simple compound strategy combining a fixed schedule with input scaling, and achieves state-of-the-art pixel-based, single-stage high-resolution ImageNet generation when used with the RIN architecture.

ABSTRACT

We empirically study the effect of noise scheduling strategies for denoising diffusion generative models. There are three findings: (1) the noise scheduling is crucial for the performance, and the optimal one depends on the task (e.g., image sizes), (2) when increasing the image size, the optimal noise scheduling shifts towards a noisier one (due to increased redundancy in pixels), and (3) simply scaling the input data by a factor of $b$ while keeping the noise schedule function fixed (equivalent to shifting the logSNR by $\log b$) is a good strategy across image sizes. This simple recipe, when combined with recently proposed Recurrent Interface Network (RIN), yields state-of-the-art pixel-based diffusion models for high-resolution images on ImageNet, enabling single-stage, end-to-end generation of diverse and high-fidelity images at 1024$ imes$1024 resolution (without upsampling/cascades).

Motivation & Objective

• Investigate how different noise scheduling strategies affect diffusion model performance across image resolutions.
• Understand why and how image size influences the effectiveness of noise schedules.
• Develop a simple, effective compound scheduling approach that is practical across tasks and resolutions.
• Show that combining the proposed scheduling with Recurrent Interface Network (RIN) yields high-quality, single-stage high-resolution image generation.

Proposed method

• Model diffusion with continuous-time noise schedules gamma(t) and analyze cosine, sigmoid, and linear forms.
• Explore input scaling b as an indirect way to adjust logSNR and its effect on training stability and performance.
• Propose a simple compound strategy: fix a noise schedule (e.g., gamma(t)=1-t) and apply input scaling, with optional variance normalization.
• Use variance normalization of x_t as a pre-processing step for the denoising network during training and sampling.
• Train and evaluate on class-conditional ImageNet across multiple resolutions using FID and Inception Score as metrics.
• Demonstrate that cosine or sigmoid schedules with tuned hyper-parameters and input scaling yield better results than a fixed linear schedule in many settings.

Experimental results

Research questions

• RQ1 How does the choice of noise schedule gamma(t) impact diffusion model performance across different image resolutions?
• RQ2 Does increasing image size necessitate a shift toward noisier schedules due to pixel redundancy?
• RQ3 Can a simple combination of fixed noise scheduling with input scaling provide robust performance across resolutions?
• RQ4 What is the impact of combining the proposed scheduling strategy with the Recurrent Interface Network (RIN) on high-resolution, pixel-based diffusion generation?

Key findings

• Noise scheduling is crucial for diffusion model performance and optimal schedules depend on the task and image size.
• As image size increases, the optimal schedule shifts toward a noisier regime due to higher pixel redundancy and independence of per-pixel noise.
• Scaling the input by a factor b, while keeping the schedule fixed, effectively adjusts logSNR and yields robust improvements across resolutions.
• A simple compound strategy (fixed gamma(t) such as 1−t plus input scaling) combined with RIN enables state-of-the-art single-stage, pixel-based high-resolution image generation on ImageNet (1024×1024) without upsampling or cascades.
• The compound strategy with RIN achieves competitive or superior FID and IS scores across multiple resolutions compared to several baselines, without relying on latent diffusion or classifier guidance.
• Visualization shows the global structure is preserved across resolutions with some finer details potentially improvable via scaling or hyper-parameter tuning.