[Paper Review] Implementing and Experimenting with Diffusion Models for Text-to-Image Generation

Robin Zbinden|arXiv (Cornell University)|Sep 22, 2022

Computational Physics and Python Applications4 citations

TL;DR

This paper presents a lightweight, publicly available implementation of a text-to-image diffusion model inspired by DALL-E 2, achieving high-quality image generation with significantly reduced computational costs. It introduces a novel image guidance method that improves image quality and alignment, enabling effective experimentation in low-resource settings while providing deeper ablation studies and analysis than prior work.

ABSTRACT

Taking advantage of the many recent advances in deep learning, text-to-image generative models currently have the merit of attracting the general public attention. Two of these models, DALL-E 2 and Imagen, have demonstrated that highly photorealistic images could be generated from a simple textual description of an image. Based on a novel approach for image generation called diffusion models, text-to-image models enable the production of many different types of high resolution images, where human imagination is the only limit. However, these models require exceptionally large amounts of computational resources to train, as well as handling huge datasets collected from the internet. In addition, neither the codebase nor the models have been released. It consequently prevents the AI community from experimenting with these cutting-edge models, making the reproduction of their results complicated, if not impossible. In this thesis, we aim to contribute by firstly reviewing the different approaches and techniques used by these models, and then by proposing our own implementation of a text-to-image model. Highly based on DALL-E 2, we introduce several slight modifications to tackle the high computational cost induced. We thus have the opportunity to experiment in order to understand what these models are capable of, especially in a low resource regime. In particular, we provide additional and analyses deeper than the ones performed by the authors of DALL-E 2, including ablation studies. Besides, diffusion models use so-called guidance methods to help the generating process. We introduce a new guidance method which can be used in conjunction with other guidance methods to improve image quality. Finally, the images generated by our model are of reasonably good quality, without having to sustain the significant training costs of state-of-the-art text-to-image models.

Motivation & Objective

• To address the lack of public access to state-of-the-art text-to-image models like DALL-E 2 and Imagen, which are computationally expensive and not open-sourced.
• To implement a functional, trainable text-to-image model based on diffusion principles, with reduced training costs compared to existing models.
• To conduct in-depth ablation studies and analyze the behavior of CLIP embeddings in text-to-image generation.
• To introduce and evaluate a new image guidance method that enhances image quality and text-image alignment.
• To demonstrate that high-quality image generation is achievable without the massive computational resources required by SOTA models.

Proposed method

• The model uses a diffusion-based image decoder trained on a curated, filtered dataset to reduce bias and computational load.
• A CLIP translator network is implemented as a simple MLP to map text embeddings to image latents, avoiding the complex prior used in DALL-E 2.
• The training pipeline leverages CLIP's text and image encoders to align textual prompts with image latents using contrastive loss.
• A novel image guidance method is introduced, which conditions the diffusion process on a reference image embedding to improve fidelity and alignment.
• The full pipeline is evaluated through ablation studies on image decoder, CLIP translator, and upsampler components.
• Experiments include qualitative and quantitative analysis of CLIP embedding semantics, including vector arithmetic operations on text and image embeddings.

Experimental results

Research questions

• RQ1 Can a lightweight, publicly available text-to-image diffusion model be implemented with significantly reduced computational cost while maintaining high image quality?
• RQ2 How do different components—such as the CLIP translator, image decoder, and upsampler—affect the quality and alignment of generated images?
• RQ3 What is the impact of the proposed image guidance method on image quality and text-image alignment compared to standard guidance techniques?
• RQ4 To what extent do CLIP embeddings exhibit semantic regularities that allow for meaningful vector arithmetic in text-to-image generation?
• RQ5 How do model and dataset scaling affect performance, and what trade-offs exist between quality and computational cost?

Key findings

• The proposed model generates reasonably high-quality images without requiring the massive computational resources of state-of-the-art models like DALL-E 2 or Imagen.
• The use of a simple MLP for the CLIP translator significantly reduces computational cost compared to the prior model in DALL-E 2, without sacrificing performance.
• The proposed image guidance method improves image quality and enhances text-image alignment when used in conjunction with other guidance techniques.
• CLIP embeddings exhibit strong semantic regularities, enabling meaningful vector arithmetic operations that allow for content manipulation and composition of different textual and visual concepts.
• Ablation studies reveal that certain image types are easier to generate, and that component-specific improvements (e.g., in upsampling or CLIP translation) yield measurable gains in image quality.
• While scaling the model and dataset improves performance, the gains are offset by increased computational cost, highlighting a key trade-off in resource-constrained settings.