[Paper Review] Diffusion Probabilistic Model Made Slim

Xingyi Yang, Daquan Zhou+2 more|arXiv (Cornell University)|Nov 27, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes Spectral Diffusion (SD), a lightweight diffusion model that enhances high-frequency detail recovery in compact architectures through wavelet gating and spectrum-aware distillation. By modeling frequency dynamics during reverse denoising and distilling high-frequency knowledge via inverse-weighted loss, SD achieves 8–18× smaller model size and 2–5× faster inference than standard latent diffusion models, with minimal fidelity drop.

ABSTRACT

Despite the recent visually-pleasing results achieved, the massive computational cost has been a long-standing flaw for diffusion probabilistic models (DPMs), which, in turn, greatly limits their applications on resource-limited platforms. Prior methods towards efficient DPM, however, have largely focused on accelerating the testing yet overlooked their huge complexity and sizes. In this paper, we make a dedicated attempt to lighten DPM while striving to preserve its favourable performance. We start by training a small-sized latent diffusion model (LDM) from scratch, but observe a significant fidelity drop in the synthetic images. Through a thorough assessment, we find that DPM is intrinsically biased against high-frequency generation, and learns to recover different frequency components at different time-steps. These properties make compact networks unable to represent frequency dynamics with accurate high-frequency estimation. Towards this end, we introduce a customized design for slim DPM, which we term as Spectral Diffusion (SD), for light-weight image synthesis. SD incorporates wavelet gating in its architecture to enable frequency dynamic feature extraction at every reverse steps, and conducts spectrum-aware distillation to promote high-frequency recovery by inverse weighting the objective based on spectrum magni tudes. Experimental results demonstrate that, SD achieves 8-18x computational complexity reduction as compared to the latent diffusion models on a series of conditional and unconditional image generation tasks while retaining competitive image fidelity.

Motivation & Objective

• To address the challenge of training compact diffusion models that fail to preserve high-frequency image details.
• To investigate why small diffusion models underperform in texture and edge recovery despite architectural pruning.
• To develop a method that explicitly models frequency dynamics and biases in the reverse denoising process for improved compact model performance.
• To achieve significant model compression and inference speedup without sacrificing image quality in conditional and unconditional generation tasks.

Proposed method

• Introduces wavelet gating in skip connections of U-Net to dynamically modulate high-, low-, and band-pass frequency components at each reverse denoising step.
• Applies a learnable gating function to wavelet-transformed features, enabling adaptive frequency response without increasing parameter count.
• Employs spectrum-aware knowledge distillation by weighting the distillation loss inversely to frequency magnitude, emphasizing high-frequency component recovery.
• Trains a small student model using a pre-trained teacher model, with loss components separated into spatial and frequency domains for targeted optimization.
• Uses discrete Fourier transform (DFT) analysis to diagnose frequency bias and evolution in generated images, validating design choices.
• Combines wavelet gating and spectrum-aware distillation into a unified training objective to preserve high-frequency details in slim models.

Figure 1 : (1) Visualizing the frequency domain gap among generated images with the full DPM [ 43 ] , Lite DPM and our SD on FFHQ [ 26 ] dataset. Lite-DPM is unable to recover fine-grained textures, while SD can produce sharp edges and realistic patterns. (2) Model size, Multiply-Add cumulation (MAC

Experimental results

Research questions

• RQ1 Why do small diffusion models fail to recover high-frequency image components like textures and edges?
• RQ2 How does the frequency content of denoising outputs evolve across reverse denoising steps in standard diffusion models?
• RQ3 To what extent does frequency bias in the denoising process impair small models' ability to reconstruct fine details?
• RQ4 Can wavelet-based gating improve frequency-aware feature representation in compact diffusion models?
• RQ5 Does inverse-weighting of frequency components in knowledge distillation enhance high-frequency detail recovery in small models?

Key findings

• Spectral Diffusion reduces model size by 8–18× and inference time by 2–5× compared to standard latent diffusion models, while maintaining competitive image quality.
• The ablation study shows that removing wavelet gating increases FID from 10.5 to 12.4 on FFHQ, proving its critical role in preserving image details.
• Spectrum-aware distillation contributes 1.8 FID points of improvement, significantly outperforming spatial-only distillation.
• With 77.6M parameters, SD achieves a FID of 18.43 on MS-COCO text-to-image generation, 18.7× smaller than the baseline LDM.
• Visualizations confirm that SD recovers fine textures like hair and architectural patterns, while baseline models produce smoother, less detailed outputs.
• The gating function values evolve over denoising steps: high-frequency components are enhanced in late-stage upsample operations, aligning with theoretical frequency evolution.

Figure 2 : Illustration of the Frequency Evolution and Bias for Diffusion Models. In the reverse process, the optimal filters recover low-frequency components first and add on the details at the end. The predicted score functions may be incorrect for rare patterns, thus failing to recover complex an