[论文解读] Efficient Diffusion Models for Vision: A Survey

Anwaar Ulhaq, Naveed Akhtar|arXiv (Cornell University)|Oct 7, 2022

Fractional Differential Equations Solutions被引用 30

一句话总结

对计算高效的扩散模型在视觉领域的综述，详细介绍在保持质量的同时加速采样的设计与过程策略。

ABSTRACT

Diffusion Models (DMs) have demonstrated state-of-the-art performance in content generation without requiring adversarial training. These models are trained using a two-step process. First, a forward - diffusion - process gradually adds noise to a datum (usually an image). Then, a backward - reverse diffusion - process gradually removes the noise to turn it into a sample of the target distribution being modelled. DMs are inspired by non-equilibrium thermodynamics and have inherent high computational complexity. Due to the frequent function evaluations and gradient calculations in high-dimensional spaces, these models incur considerable computational overhead during both training and inference stages. This can not only preclude the democratization of diffusion-based modelling, but also hinder the adaption of diffusion models in real-life applications. Not to mention, the efficiency of computational models is fast becoming a significant concern due to excessive energy consumption and environmental scares. These factors have led to multiple contributions in the literature that focus on devising computationally efficient DMs. In this review, we present the most recent advances in diffusion models for vision, specifically focusing on the important design aspects that affect the computational efficiency of DMs. In particular, we emphasize the recently proposed design choices that have led to more efficient DMs. Unlike the other recent reviews, which discuss diffusion models from a broad perspective, this survey is aimed at pushing this research direction forward by highlighting the design strategies in the literature that are resulting in practicable models for the broader research community. We also provide a future outlook of diffusion models in vision from their computational efficiency viewpoint.

研究动机与目标

• 因高计算和能源成本，激励对高效扩散模型的需求。
• 对在视觉扩散模型中提升效率的设计选择和过程策略进行分类与综合。
• 突出可使扩散式视觉系统更快、更加易用的实际设计模式。
• 就扩散模型中面向效率的研究方向提供前瞻性视角。

提出的方法

• 回顾扩散模型基础及三种与效率相关的影响力架构（DDPM、LDM、Frido）。
• 将效率策略分为高效设计策略（EDS）和高效过程策略（EPS）。
• 在表格中将代表性工作映射到架构类别与策略类型。
• 解释引导、离散化、基于分数的方法、金字塔/多尺度方法，以及潜在空间扩散作为效率杠杆。

实验结果

研究问题

• RQ1 在视觉扩散模型中，哪些设计选择最有效地降低计算量？
• RQ2 哪些过程层面的技术在不牺牲样本质量的前提下最显著地加速采样？
• RQ3 潜在空间与多尺度方法与像素空间扩展在效率和质量方面的比较如何？
• RQ4 在高效扩散方法中，速度与保真度之间的实际权衡是什么？

主要发现

• 高效扩散工作被组织为设计策略（EDS）和过程策略（EPS）。
• 潜在扩散和多尺度（金字塔）方法通过在潜在空间或跨尺度上操作显著提高效率。
• 引导策略（分类器引导与无分类器引导）影响保真度和多样性，常以牺牲多样性换取质量。
• 各种采样加速（基于SDE、ODE求解器和快速采样技术）相比普通 DDPM 能实现显著加速。
• 金字塔和潜在空间设计（如 LDM、Frido）在每个样本上减少计算量，同时保持高视觉质量。
• 本综述指出扩散效率与 GAN 之间仍存在差距，但强调快速进展使实际扩散模型成为可能。