[Paper Review] Denoising Diffusion Step-aware Models

Shuai Yang, Yukang Chen+3 more|arXiv (Cornell University)|Oct 5, 2023

Generative Adversarial Networks and Image SynthesisComputer Science3 citations

TL;DR

This paper proposes Denoising Diffusion Step-aware Models (DDSM), a framework that dynamically adjusts neural network size across denoising steps using evolutionary search to reduce computational cost. By assigning smaller, pruned networks to less critical steps and larger ones to key stages, DDSM achieves up to 76% FLOPs reduction across datasets like ImageNet and CelebA-HQ without sacrificing sample quality.

ABSTRACT

Denoising Diffusion Probabilistic Models (DDPMs) have garnered popularity for data generation across various domains. However, a significant bottleneck is the necessity for whole-network computation during every step of the generative process, leading to high computational overheads. This paper presents a novel framework, Denoising Diffusion Step-aware Models (DDSM), to address this challenge. Unlike conventional approaches, DDSM employs a spectrum of neural networks whose sizes are adapted according to the importance of each generative step, as determined through evolutionary search. This step-wise network variation effectively circumvents redundant computational efforts, particularly in less critical steps, thereby enhancing the efficiency of the diffusion model. Furthermore, the step-aware design can be seamlessly integrated with other efficiency-geared diffusion models such as DDIMs and latent diffusion, thus broadening the scope of computational savings. Empirical evaluations demonstrate that DDSM achieves computational savings of 49% for CIFAR-10, 61% for CelebA-HQ, 59% for LSUN-bedroom, 71% for AFHQ, and 76% for ImageNet, all without compromising the generation quality.

Motivation & Objective

• Address the high computational cost of denoising diffusion models, which require full-network inference at every step during generation.
• Investigate whether all denoising steps require equal computational resources, challenging the assumption of uniform model complexity across steps.
• Develop a method to adaptively assign network capacity based on the importance of each step, minimizing redundant computation.
• Ensure compatibility with existing acceleration techniques like DDIM and latent diffusion to broaden practical applicability.
• Demonstrate that dataset-specific step importance distributions significantly affect model efficiency and performance.

Proposed method

• Propose a step-aware architecture where neural network capacity varies per denoising step, using pruned versions of a base U-Net.
• Employ evolutionary search to determine the optimal network size (from small to large) for each step based on generation quality and computational cost.
• Prune the full-precision U-Net into multiple lightweight variants that share weights with the original, avoiding retraining.
• Integrate the step-aware design with existing diffusion acceleration methods such as DDIM and latent diffusion for further efficiency gains.
• Use a smoothed color bar and line graph visualization to represent the search results, showing model size distribution across steps.
• Evaluate the method across five diverse datasets to validate generalization and dataset-dependent optimization strategies.

Experimental results

Research questions

• RQ1 Do all denoising steps in diffusion models require the same level of computational complexity?
• RQ2 Can dynamic network pruning across steps reduce FLOPs without degrading sample quality?
• RQ3 How does the importance of each step vary across different datasets with distinct structural and content characteristics?
• RQ4 To what extent can step-aware model pruning be combined with other acceleration techniques like DDIM or latent diffusion?
• RQ5 Is there a dataset-specific pattern in the importance distribution of denoising steps, and can it be leveraged for better efficiency?

Key findings

• DDSM achieves a 76% reduction in FLOPs for ImageNet generation while maintaining high-quality samples, demonstrating the highest efficiency gain among all evaluated datasets.
• The method reduces computational cost by 61% for CelebA-HQ, 59% for LSUN-bedroom, 49% for CIFAR-10, and 71% for AFHQ, with consistent performance across diverse data distributions.
• The search results show that early steps in CelebA-HQ require larger models due to strict structural constraints (e.g., facial symmetry), whereas CIFAR-10 and ImageNet use smaller models early on due to higher structural variability.
• Swapping the optimal strategy between datasets leads to significant performance drops—e.g., using CIFAR-10’s strategy on CelebA increases FID from 6.039 to 7.431—confirming the necessity of dataset-specific search.
• The step-aware design is orthogonal to existing acceleration methods and can be seamlessly combined with DDIM and latent diffusion, further boosting efficiency.
• The method reveals that detail generation in high-resolution face datasets like CelebA-HQ is less sensitive to model capacity, while structure generation in diverse datasets like CIFAR-10 and ImageNet demands higher capacity in early steps.