[Paper Review] Prompt-to-Prompt Image Editing with Cross Attention Control

Amir Hertz, Ron Mokady+4 more|arXiv (Cornell University)|Aug 2, 2022

Modular Robots and Swarm Intelligence361 citations

TL;DR

The paper presents a text-only Prompt-to-Prompt editing framework that manipulates cross-attention maps inside a diffusion model to edit images by editing prompts, preserving original structure without masks.

ABSTRACT

Recent large-scale text-driven synthesis models have attracted much attention thanks to their remarkable capabilities of generating highly diverse images that follow given text prompts. Such text-based synthesis methods are particularly appealing to humans who are used to verbally describe their intent. Therefore, it is only natural to extend the text-driven image synthesis to text-driven image editing. Editing is challenging for these generative models, since an innate property of an editing technique is to preserve most of the original image, while in the text-based models, even a small modification of the text prompt often leads to a completely different outcome. State-of-the-art methods mitigate this by requiring the users to provide a spatial mask to localize the edit, hence, ignoring the original structure and content within the masked region. In this paper, we pursue an intuitive prompt-to-prompt editing framework, where the edits are controlled by text only. To this end, we analyze a text-conditioned model in depth and observe that the cross-attention layers are the key to controlling the relation between the spatial layout of the image to each word in the prompt. With this observation, we present several applications which monitor the image synthesis by editing the textual prompt only. This includes localized editing by replacing a word, global editing by adding a specification, and even delicately controlling the extent to which a word is reflected in the image. We present our results over diverse images and prompts, demonstrating high-quality synthesis and fidelity to the edited prompts.

Motivation & Objective

• Motivation to enable intuitive, text-based image editing without user-provided masks or additional training.
• Investigation of cross-attention layers as a semantic bridge between prompt tokens and image regions.
• Development of prompt-driven editing operations that alter or preserve image structure using attention manipulation.

Proposed method

• Analyze cross-attention in text-conditioned diffusion models to link prompt tokens with spatial image regions.
• Inject and override attention maps from a source image during diffusion with an edited prompt to preserve layout.
• Define editing operations (word swap, adding a phrase, attention re-weighting) via controlled attention injection across diffusion steps.
• Provide an iterative diffusion-based algorithm that runs edits for source and edited prompts with shared randomness to maintain structure.
• Introduce mechanisms to handle partial token alignment and timestamped injection to balance fidelity and editability.

Experimental results

Research questions

• RQ1 How can cross-attention maps in text-to-image diffusion models be leveraged to control spatial layout during editing?
• RQ2 Can prompts be edited to achieve localized or global image edits without masks or retraining?
• RQ3 What are effective strategies (e.g., injection timing, soft constraints) to preserve original composition while applying prompt changes?
• RQ4 How does the proposed method perform on real images via inversion and what are its limitations?
• RQ5 What is the trade-off between fidelity to the edited prompt and retention of source image structure?

Key findings

• Cross-attention maps tightly couple pixels to prompt words and can be manipulated to control image layout.
• Injecting source attention maps into edited prompts preserves composition while enabling semantic changes.
• Soft and partial attention injection (via a timestamp tau) mitigates over-constraint and maintains editability.
• The method supports word swaps, addition of new phrases, and attention re-weighting for fine-grained control.
• Preliminary results show real-image editing via inversion is possible, with mask-based refinements addressing reconstruction gaps.