[Paper Review] eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

Yogesh Balaji, Seungjun Nah+11 more|arXiv (Cornell University)|Nov 2, 2022

Generative Adversarial Networks and Image Synthesis223 citations

TL;DR

eDiff-I trains an ensemble of expert denoisers specialized for different stages of diffusion-based text-to-image generation, improving text alignment without increasing inference cost, and leverages multiple encoders and a paint-with-words capability.

ABSTRACT

Large-scale diffusion-based generative models have led to breakthroughs in text-conditioned high-resolution image synthesis. Starting from random noise, such text-to-image diffusion models gradually synthesize images in an iterative fashion while conditioning on text prompts. We find that their synthesis behavior qualitatively changes throughout this process: Early in sampling, generation strongly relies on the text prompt to generate text-aligned content, while later, the text conditioning is almost entirely ignored. This suggests that sharing model parameters throughout the entire generation process may not be ideal. Therefore, in contrast to existing works, we propose to train an ensemble of text-to-image diffusion models specialized for different synthesis stages. To maintain training efficiency, we initially train a single model, which is then split into specialized models that are trained for the specific stages of the iterative generation process. Our ensemble of diffusion models, called eDiff-I, results in improved text alignment while maintaining the same inference computation cost and preserving high visual quality, outperforming previous large-scale text-to-image diffusion models on the standard benchmark. In addition, we train our model to exploit a variety of embeddings for conditioning, including the T5 text, CLIP text, and CLIP image embeddings. We show that these different embeddings lead to different behaviors. Notably, the CLIP image embedding allows an intuitive way of transferring the style of a reference image to the target text-to-image output. Lastly, we show a technique that enables eDiff-I's "paint-with-words" capability. A user can select the word in the input text and paint it in a canvas to control the output, which is very handy for crafting the desired image in mind. The project page is available at https://deepimagination.cc/eDiff-I/

Motivation & Objective

• Motivate the need to capture distinct synthesis stages in diffusion-based text-to-image generation.
• Propose an ensemble of expert denoisers specialized for different noise levels to improve text alignment and maintain inference cost.
• Investigate the benefit of multiple conditioning encoders (T5, CLIP text, CLIP image) for conditioning diversity.
• Introduce a training-efficient finetuning strategy to grow the ensemble without prohibitive training cost.
• Present a training-free paint-with-words mechanism for spatial control of generation.

Proposed method

• Train a base diffusion model and progressively branch it into specialized expert denoisers corresponding to noise-level intervals.
• Use a binary-tree branching scheme to initialize and fine-tune expert models with respect to split noise distributions, focusing on extreme (high/low) and intermediate intervals.
• Combine multiple input embeddings (T5 text, CLIP text, CLIP image) with cross-attention and dropout to form diverse conditioning.
• Incorporate a training-free paint-with-words mechanism by modulating cross-attention with user-drawn masks to control spatial layout.
• Deploy a cascade of diffusion models (base 64x64, SR256, SR1024) with training-time degradation to improve generalization in super-resolution stages.
• Evaluate using zero-shot FID-CLIP trade-offs on COCO and Visual Genome, and compare against state-of-the-art baselines.

Experimental results

Research questions

• RQ1 Does an ensemble of expert denoisers improve text-to-image alignment without increasing inference cost?
• RQ2 How do multiple conditioning encoders (T5, CLIP text, CLIP image) affect image quality and style transfer capabilities?
• RQ3 Can a training-free paint-with-words mechanism provide practical spatial control over generated outputs?
• RQ4 What is the performance gain of eDiff-I versus single-model baselines on standard text-to-image benchmarks?

Key findings

• The 2-Expert-Ensemble consistently improves the FID-CLIP trade-off over the baseline across COCO and Visual Genome datasets.
• eDiff-I achieves competitive zero-shot FID while maintaining inference cost comparable to single-model diffusion.
• Using a combination of T5 and CLIP text encoders yields best performance, with CLIP image embeddings enabling style transfer.
• Paint-with-words provides spatial control by modulating cross-attention with user-provided masks, without additional training.
• Training-efficient branching (starting from a shared base, then finetuning left/right/high-noise extremes and a middle expert) reduces training cost while expanding capacity.
• Compared against large baselines, eDiff-I variants (Config A-D) achieve progressively better zero-shot FID, with Config D reaching 6.95 zero-shot FID for the reported setup.