[Paper Review] On the Design Fundamentals of Diffusion Models: A Survey

Ziyi Chang, George Alex Koulieris+2 more|arXiv (Cornell University)|Jun 7, 2023

Statistical Methods and Bayesian InferenceMathematics46 citations

TL;DR

This paper provides a comprehensive, component-wise survey of diffusion models, detailing design choices for the forward process, reverse process, and sampling procedure, in both discrete and continuous formulations.

ABSTRACT

Diffusion models are learning pattern-learning systems to model and sample from data distributions with three functional components namely the forward process, the reverse process, and the sampling process. The components of diffusion models have gained significant attention with many design factors being considered in common practice. Existing reviews have primarily focused on higher-level solutions, covering less on the design fundamentals of components. This study seeks to address this gap by providing a comprehensive and coherent review of seminal designable factors within each functional component of diffusion models. This provides a finer-grained perspective of diffusion models, benefiting future studies in the analysis of individual components, the design factors for different purposes, and the implementation of diffusion models.

Motivation & Objective

• Organize diffusion-model design fundamentals around the three core components: forward process, reverse process, and sampling procedure.
• Provide a fine-grained, component-level review to aid analysis, applicability, and implementation of diffusion models.
• Compare discrete and continuous formulations and discuss how data properties and noise design affect learning and generation.

Proposed method

• Survey and synthesize literature on diffusion-model design choices for the forward process, including noise schedule, noise type, and transition chains.
• Describe the reverse process architecture and output parameterizations, notably U-Net and Transformer-based denoisers, and how they are trained.
• Explain the sampling procedure and how pretrained denoisers generate new data, including different output parameterizations (x0, epsilon, score).
• Present the continuous-time (SDE) and discrete-time (DDPM) formulations and their implications for theory and practice.
• Discuss data properties, latent spaces, and systematic transitions as design levers to improve expressiveness and efficiency.

Experimental results

Research questions

• RQ1 What are the standard design choices for the forward process and how do they impact learning and generation?
• RQ2 How should the reverse process be parameterized and trained to accurately denoise across timesteps?
• RQ3 What are the effects of discrete versus continuous timesteps on diffusion modeling theory and practice?
• RQ4 How do noise schedules, noise types, and transition chains influence model performance and convergence?
• RQ5 How can data properties and latent representations be leveraged to enhance diffusion model expressiveness?

Key findings

• The forward process adds noise through a time-ordered chain with no trainable parameters, leading to a tractable terminal distribution.
• Discrete DDPMs and continuous SDE formulations provide equivalent ends via different mathematical representations, guiding training and sampling.
• The noise schedule and noise type critically shape exploration vs. exploitation and model expressiveness.
• Reverse process is typically modeled as a Gaussian with learnable mean and fixed or learned variance, parameterized via outputs like x̂0, ε̂t, or ŝt.
• Common denoising architectures include U-Net and Transformer, with hybrids leveraging skip connections and cross-attention to balance local and global dependencies.
• Sampling uses the trained denoiser to reverse the forward chain, yielding samples that approximate the data distribution.