[Paper Review] Efficient Diffusion Models for Vision: A Survey

Anwaar Ulhaq, Naveed Akhtar|arXiv (Cornell University)|Oct 7, 2022

Fractional Differential Equations Solutions30 citations

TL;DR

A survey of computationally efficient diffusion models for vision, detailing design and process strategies to speed up sampling while preserving quality.

ABSTRACT

Diffusion Models (DMs) have demonstrated state-of-the-art performance in content generation without requiring adversarial training. These models are trained using a two-step process. First, a forward - diffusion - process gradually adds noise to a datum (usually an image). Then, a backward - reverse diffusion - process gradually removes the noise to turn it into a sample of the target distribution being modelled. DMs are inspired by non-equilibrium thermodynamics and have inherent high computational complexity. Due to the frequent function evaluations and gradient calculations in high-dimensional spaces, these models incur considerable computational overhead during both training and inference stages. This can not only preclude the democratization of diffusion-based modelling, but also hinder the adaption of diffusion models in real-life applications. Not to mention, the efficiency of computational models is fast becoming a significant concern due to excessive energy consumption and environmental scares. These factors have led to multiple contributions in the literature that focus on devising computationally efficient DMs. In this review, we present the most recent advances in diffusion models for vision, specifically focusing on the important design aspects that affect the computational efficiency of DMs. In particular, we emphasize the recently proposed design choices that have led to more efficient DMs. Unlike the other recent reviews, which discuss diffusion models from a broad perspective, this survey is aimed at pushing this research direction forward by highlighting the design strategies in the literature that are resulting in practicable models for the broader research community. We also provide a future outlook of diffusion models in vision from their computational efficiency viewpoint.

Motivation & Objective

• Motivate the need for efficient diffusion models due to high computational and energy costs.
• Categorize and synthesize design choices and process strategies that improve efficiency in vision diffusion models.
• Highlight practical design patterns enabling faster, more accessible diffusion-based vision systems.
• Offer a forward-looking perspective on efficiency-oriented research directions in diffusion models.

Proposed method

• Review diffusion model fundamentals and three influential architectures (DDPM, LDM, Frido) relevant to efficiency.
• Categorize efficiency strategies into Efficient Design Strategies (EDS) and Efficient Process Strategies (EPS).
• Map representative works to architectural categories and strategy types in tables.
• Explain guidance, discretization, score-based methods, pyramidal/multi-scale approaches, and latent-space diffusion as efficiency levers.

Experimental results

Research questions

• RQ1 What design choices most effectively reduce computation in diffusion models for vision?
• RQ2 What process-level techniques most strongly accelerate sampling without sacrificing sample quality?
• RQ3 How do latent and multi-scale approaches compare to pixel-space diffusion in efficiency and quality?
• RQ4 What are the practical trade-offs between speed and fidelity in efficient diffusion methods?

Key findings

• Efficient diffusion work is organized into design strategies (EDS) and process strategies (EPS).
• Latent diffusion and multi-scale (pyramidal) approaches substantially improve efficiency by operating in latent spaces or across scales.
• Guidance strategies (classifier-guided vs classifier-free) influence fidelity and diversity, often trading off diversity for quality.
• Various sampling accelerations (SDE-based, ODE solvers, and fast sampling techniques) achieve substantial speedups over vanilla DDPMs.
• Pyramidal and latent-space designs (e.g., LDM, Frido) reduce computation per sample while maintaining high visual quality.
• The survey notes ongoing gap between diffusion efficiency and GANs, but highlights rapid advances enabling practical diffusion models.