[Paper Review] A Survey of Diffusion Based Image Generation Models: Issues and Their Solutions

Tianyi Zhang, Zheng Wang+3 more|arXiv (Cornell University)|Aug 25, 2023

FinTech, Crowdfunding, Digital Finance8 citations

TL;DR

The paper analyzes diffusion-based image generation, identifies key challenges (multi-object generation, rare concepts, and quality) and surveys solutions and techniques to address them.

ABSTRACT

Recently, there has been significant progress in the development of large models. Following the success of ChatGPT, numerous language models have been introduced, demonstrating remarkable performance. Similar advancements have also been observed in image generation models, such as Google's Imagen model, OpenAI's DALL-E 2, and stable diffusion models, which have exhibited impressive capabilities in generating images. However, similar to large language models, these models still encounter unresolved challenges. Fortunately, the availability of open-source stable diffusion models and their underlying mathematical principles has enabled the academic community to extensively analyze the performance of current image generation models and make improvements based on this stable diffusion framework. This survey aims to examine the existing issues and the current solutions pertaining to image generation models.

Motivation & Objective

• Identify main challenges in diffusion-based image generation (multi-object rendering, rare concepts, and quality) and motivate targeted solutions.
• Organize diffusion models by the specific problems they address in image generation.
• Summarize core techniques, architectures, and training/inference strategies that improve generation quality and control.
• Highlight differences from prior surveys by focusing on image generation applications and practical limitations/solutions.

Proposed method

• Review diffusion model fundamentals including forward/backward processes, DDPM, and classifier-free guidance (CFG).
• Discuss noise prediction U-net with attention and cross-attention mechanisms and their role in conditioning.
• Summarize conditioning and layout integration methods (bounding boxes, segmentation maps) and plug-in architectures like ControlNet and variants.
• Describe retrieval-based and subject-driven generation approaches to handle rare/unseen concepts and personalization.
• Outline quality improvement strategies focusing on text encoder improvements, mixture-of-experts, and prompt/sampling techniques.

Figure 1. The directed graphical model considered in DDPM (Ho et al . , 2020 ) .

Experimental results

Research questions

• RQ1 What are the main limitations of diffusion-based image generation in practice?
• RQ2 What techniques exist to incorporate layout and attention control to improve multi-object generation?
• RQ3 How can retrieval and subject-driven approaches help generate rare or unseen concepts?
• RQ4 What strategies most effectively improve image generation quality beyond mere model scaling?
• RQ5 How do different text encoders influence diffusion-based image generation quality?

Key findings

• Layout- and attention-guided methods (e.g., ControlNet, GLIGEN, Attend-and-Excite) improve multi-object rendering and layout adherence.
• Retrieval-based and subject-driven approaches enable rare/unseen concept generation and personalization with varying degrees of fine-tuning.
• Improvements in text encoders (e.g., T5-XXL, CLIP-based vs pure language models) and mixed encoder setups can surpass larger U-nets for quality gains.
• classifier-free guidance remains a foundational conditioning approach across major diffusion models.
• Techniques modifying cross-attention maps or latent optimization can enhance image quality without extensive model retraining.

Figure 2. Failed cases for multiple objects generation.