[论文解读] State of the Art on Diffusion Models for Visual Computing

Ryan Po, Wang Yi-fan+16 more|arXiv (Cornell University)|Oct 11, 2023

Mathematical Biology Tumor Growth被引用 7

一句话总结

This STAR surveys the theory, practice, and applications of diffusion models for visual computing, covering 2D to 4D generation/editing and highlighting conditioning, inversion, personalization, datasets, metrics, challenges, and societal implications.

ABSTRACT

The field of visual computing is rapidly advancing due to the emergence of generative artificial intelligence (AI), which unlocks unprecedented capabilities for the generation, editing, and reconstruction of images, videos, and 3D scenes. In these domains, diffusion models are the generative AI architecture of choice. Within the last year alone, the literature on diffusion-based tools and applications has seen exponential growth and relevant papers are published across the computer graphics, computer vision, and AI communities with new works appearing daily on arXiv. This rapid growth of the field makes it difficult to keep up with all recent developments. The goal of this state-of-the-art report (STAR) is to introduce the basic mathematical concepts of diffusion models, implementation details and design choices of the popular Stable Diffusion model, as well as overview important aspects of these generative AI tools, including personalization, conditioning, inversion, among others. Moreover, we give a comprehensive overview of the rapidly growing literature on diffusion-based generation and editing, categorized by the type of generated medium, including 2D images, videos, 3D objects, locomotion, and 4D scenes. Finally, we discuss available datasets, metrics, open challenges, and social implications. This STAR provides an intuitive starting point to explore this exciting topic for researchers, artists, and practitioners alike.

研究动机与目标

• Introduce the fundamental theory and mathematics of diffusion models as applied to visual computing.
• Provide a structured, media-oriented overview of diffusion-based generation and editing across 2D, video, 3D, and 4D data.
• Discuss data availability, evaluation metrics, and practical design choices impacting diffusion models.
• Highlight open challenges and societal implications to guide future research and responsible use.

提出的方法

• Present the diffusion process and score-based denoising framework as the core mathematical foundation.
• Use latent diffusion models (LDMs) to reduce computational cost by operating in a latent space with an encoder–decoder architecture.
• Explain conditioning and guidance mechanisms, including cross-attention, concatenation, and classifier-free guidance.
• Describe editing, inversion, and customization techniques enabling manipulation and personalization of outputs.
• Summarize diffusion model applications across 2D images, video, 3D objects/scenes, and 4D spatiotemporal data, with discussion of datasets and metrics.

Figure 1 : Diffusion Process. (A) The forward SDE transforms images to noise. The forward SDE can be reversed [ And82 ] if we can predict the score function, enabling image synthesis. (B) The distributions of images and noise are linked with stochastic trajectories, modeled by SDEs, and deterministi

实验结果

研究问题

• RQ1 What are the essential mathematical foundations and practical design choices of diffusion models for visual computing?
• RQ2 How do conditioning and guidance mechanisms enable controllable generation across 2D, video, 3D, and 4D content?
• RQ3 What are effective editing, inversion, and customization techniques for diffusion-based workflows?
• RQ4 What datasets, metrics, open challenges, and societal implications shape current and future diffusion-model systems?

主要发现

• Diffusion models have become the de-facto standard for generating and editing images, videos, 3D objects, and 4D scenes in visual computing.
• Latent diffusion models reduce computational cost by operating in compressed latent spaces while preserving perceptual quality.
• Conditioning via cross-attention and guidance methods (including classifier-free guidance) provide flexible control over outputs and trade-offs between diversity and quality.
• Editing and inversion techniques (e.g., DDIM inversion, textual inversion, and DreamBooth-style customization) enable targeted manipulation and personalization.
• The STAR discusses datasets, evaluation metrics, open challenges, and social implications, highlighting the rapid growth and need for responsible development.

Figure 2 : Stable Diffusion. This schematic shows an overview of the latent diffusion approach, including encoder $\mathcal{E}$ , decoder $\mathcal{D}$ , and conditioning using a cross-attention mechanism. Figure adapted from [ RBL ∗ 22 ] .