[Paper Review] Generative AI in Vision: A Survey on Models, Metrics and Applications

Gaurav Raut, Apoorv Singh|arXiv (Cornell University)|Feb 26, 2024

Retinal Imaging and AnalysisMedicine3 citations

TL;DR

This survey provides a comprehensive overview of diffusion and legacy generative models in computer vision, detailing their theoretical foundations, state-of-the-art architectures, and diverse applications such as text-to-image generation, image inpainting, and super-resolution. It highlights key advancements in denoising diffusion probabilistic models (DDPM), score-based modeling, and latent diffusion, while identifying critical challenges and future research directions in scalability, interpretability, and ethical AI.

ABSTRACT

Generative AI models have revolutionized various fields by enabling the creation of realistic and diverse data samples. Among these models, diffusion models have emerged as a powerful approach for generating high-quality images, text, and audio. This survey paper provides a comprehensive overview of generative AI diffusion and legacy models, focusing on their underlying techniques, applications across different domains, and their challenges. We delve into the theoretical foundations of diffusion models, including concepts such as denoising diffusion probabilistic models (DDPM) and score-based generative modeling. Furthermore, we explore the diverse applications of these models in text-to-image, image inpainting, and image super-resolution, along with others, showcasing their potential in creative tasks and data augmentation. By synthesizing existing research and highlighting critical advancements in this field, this survey aims to provide researchers and practitioners with a comprehensive understanding of generative AI diffusion and legacy models and inspire future innovations in this exciting area of artificial intelligence.

Motivation & Objective

• To provide a systematic review of generative AI models in computer vision, focusing on diffusion and legacy architectures.
• To analyze the theoretical underpinnings of diffusion models, including DDPM and score-based generative modeling.
• To survey state-of-the-art applications such as text-to-image synthesis, image inpainting, super-resolution, and anomaly detection.
• To identify key challenges in training stability, scalability, and interpretability of diffusion models.
• To outline future research directions, including time-series forecasting, physics-inspired modeling, and ethical considerations.

Proposed method

• Surveying foundational generative models including VAEs, GANs, normalizing flows, and energy-based models.
• Detailing the forward and reverse diffusion processes in DDPM, where noise is gradually added and then removed via learned denoising.
• Explaining the use of Markov chains in the forward process: $ x_t = \sqrt{1-\beta_t}x_{t-1} + \sqrt{\beta_t}\epsilon $, with $ \epsilon \sim \mathcal{N}(0, I) $.
• Presenting the reverse diffusion kernel (RDK) as a neural network that predicts noise at each step to reconstruct data.
• Reviewing conditional diffusion models like Stable Diffusion and Palette, which use text or masked image conditioning.
• Analyzing specialized applications such as AnoDDPM for anomaly detection using DDPM-based reconstruction with simplex noise perturbation.

Figure 1 : a) Images generated using stable diffusion [ 70 ] ; b) Image super-resolution results from SR3 [ 72 ] ; c) Image inpainting results from Palette [ 73 ]

Experimental results

Research questions

• RQ1 How do diffusion models like DDPM and score-based models achieve high-fidelity image generation compared to GANs and VAEs?
• RQ2 What are the key architectural and training innovations that enable state-of-the-art performance in text-to-image and image inpainting tasks?
• RQ3 How can diffusion models be adapted for downstream tasks such as image super-resolution, colorization, and anomaly detection?
• RQ4 What are the main limitations of current diffusion models in terms of training stability, inference speed, and interpretability?
• RQ5 What future research directions—such as physics-inspired modeling or time-series forecasting—hold the most promise for advancing diffusion-based generative AI?

Key findings

• Diffusion models, particularly DDPM and latent diffusion, achieve high-quality image generation by learning to reverse a gradual noise addition process.
• Conditional diffusion models like Palette and RePaint enable high-fidelity image inpainting by denoising only the masked regions using Markov chain properties.
• AnoDDPM achieves anomaly detection by training a DDPM to reconstruct healthy images and measuring reconstruction error on perturbed test samples.
• Latent diffusion models such as Stable Diffusion significantly improve efficiency by operating in a compressed latent space.
• Despite strong performance, diffusion models face challenges in training stability, computational cost, and interpretability.
• Future research is needed in time-series forecasting, physics-inspired modeling, and addressing ethical concerns like bias and societal impact.

Figure 2 : An extension of generative models classification based on [ 25 ]