[Paper Review] Energy-Based Cross Attention for Bayesian Context Update in Text-to-Image Diffusion Models

Geon Yeong Park, Jeongsol Kim+3 more|arXiv (Cornell University)|Jun 16, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes Energy-Based Cross Attention (EBCA), a training-free, Bayesian framework that improves semantic alignment in text-to-image diffusion models by adaptively updating context vectors through energy-based posterior estimation. By minimizing hierarchical energy functions in cross-attention layers, EBCA enables zero-shot compositional generation and significantly reduces semantic misalignment in multi-concept generation, inpainting, and image editing tasks without finetuning.

ABSTRACT

Despite the remarkable performance of text-to-image diffusion models in image generation tasks, recent studies have raised the issue that generated images sometimes cannot capture the intended semantic contents of the text prompts, which phenomenon is often called semantic misalignment. To address this, here we present a novel energy-based model (EBM) framework for adaptive context control by modeling the posterior of context vectors. Specifically, we first formulate EBMs of latent image representations and text embeddings in each cross-attention layer of the denoising autoencoder. Then, we obtain the gradient of the log posterior of context vectors, which can be updated and transferred to the subsequent cross-attention layer, thereby implicitly minimizing a nested hierarchy of energy functions. Our latent EBMs further allow zero-shot compositional generation as a linear combination of cross-attention outputs from different contexts. Using extensive experiments, we demonstrate that the proposed method is highly effective in handling various image generation tasks, including multi-concept generation, text-guided image inpainting, and real and synthetic image editing. Code: https://github.com/EnergyAttention/Energy-Based-CrossAttention.

Motivation & Objective

• To address semantic misalignment in text-to-image diffusion models, where generated images fail to reflect intended textual concepts.
• To improve context vector quality by modeling the posterior distribution $ p(\text{context} \mid \text{representations}) $, rather than relying on fixed, frozen text embeddings.
• To enable zero-shot compositional generation through the inherent compositionality of Energy-Based Models (EBMs).
• To provide a plug-and-play, training-free solution compatible with existing diffusion models like Stable Diffusion.
• To demonstrate effectiveness across diverse tasks including multi-concept generation, text-guided inpainting, and real/synthetic image editing.

Proposed method

• Formulates energy functions $ E(K_l; Q_{t,l}) $ in each cross-attention layer to model the posterior $ p(K_l \mid Q_{t,l}) $, where $ K_l $ is the key (context) and $ Q_{t,l} $ is the query (image representation).
• Derives the gradient of the log posterior of context vectors to enable iterative, adaptive updates that propagate through subsequent layers.
• Introduces a prior energy function $ \operatorname{E}(K) $ that adaptively regularizes the smooth maximum of $ \|k_i\| $, preventing over-penalization of context vectors.
• Uses a single-step context update with propagation to subsequent layers, balancing performance and computational cost.
• Enables zero-shot compositional generation via linear combination of cross-attention outputs from multiple energy-based context vectors.
• Applies the framework in inference only, without any model fine-tuning or additional training.

Figure 1: The energy-based cross-attention improves the semantic alignment between given text and the generated sample. The proposed method could be leveraged for multiple applications without additional training.

Experimental results

Research questions

• RQ1 Can energy-based posterior estimation improve semantic alignment between text prompts and generated images in diffusion models?
• RQ2 How does adaptive context updating through energy function minimization affect multi-concept generation and image editing?
• RQ3 Can the proposed method enable zero-shot compositional generation without retraining?
• RQ4 Does the adaptive prior energy function prevent over-suppression of salient context vectors compared to uniform regularization?
• RQ5 How does the number of context updates affect image quality and semantic fidelity?

Key findings

• The proposed EBCA method significantly reduces semantic misalignment, as evidenced by improved generation quality in multi-concept tasks such as 'a lion and a crown'.
• With increasing regularization strength $ \alpha $, salient contextual elements (e.g., crown, lion) progressively vanish, confirming the adaptive penalization's effectiveness in preserving key concepts.
• Using the final timestep context vector $ C_T $ as a fixed embedding yields semantically accurate generations, demonstrating that the adaptive update preserves intended meaning.
• Multi-step context updates improve image quality but are computationally expensive, indicating that a single-step update is sufficient for optimal performance.
• The method enables effective real and synthetic image editing, including editing horses to zebras and cats to dogs, using only text prompts and no training.
• The framework supports zero-shot compositional editing by linearly combining cross-attention outputs from multiple energy-based context vectors, enabling flexible and intuitive manipulation.

Figure 2: Comparison between the Stable-Diffusion and the proposed method. ( A ) The Stable-Diffusion uses fixed context embedding encoded by pre-trained CLIP. ( B ) The proposed method allows adaptive context embedding through energy-based context update (EBCU) and energy-based composition of queri