[Paper Review] LEDITS: Real Image Editing with DDPM Inversion and Semantic Guidance

Linoy Tsaban, Apolinário Passos|arXiv (Cornell University)|Jul 2, 2023

Generative Adversarial Networks and Image Synthesis16 references8 citations

TL;DR

LEDITS combines DDPM inversion with SEGA semantic guidance to edit real images in a lightweight, flexible way without changing model architecture.

ABSTRACT

Recent large-scale text-guided diffusion models provide powerful image-generation capabilities. Currently, a significant effort is given to enable the modification of these images using text only as means to offer intuitive and versatile editing. However, editing proves to be difficult for these generative models due to the inherent nature of editing techniques, which involves preserving certain content from the original image. Conversely, in text-based models, even minor modifications to the text prompt frequently result in an entirely distinct result, making attaining one-shot generation that accurately corresponds to the users intent exceedingly challenging. In addition, to edit a real image using these state-of-the-art tools, one must first invert the image into the pre-trained models domain - adding another factor affecting the edit quality, as well as latency. In this exploratory report, we propose LEDITS - a combined lightweight approach for real-image editing, incorporating the Edit Friendly DDPM inversion technique with Semantic Guidance, thus extending Semantic Guidance to real image editing, while harnessing the editing capabilities of DDPM inversion as well. This approach achieves versatile edits, both subtle and extensive as well as alterations in composition and style, while requiring no optimization nor extensions to the architecture.

Motivation & Objective

• Motivate real-image editing with text-guided diffusion models and address editing challenges.
• Propose a lightweight integration of DDPM inversion with SEGA to enable semantic-guided edits on real images.
• Show that combining DDPM inversion and SEGA yields versatile edits while preserving fidelity and semantic control.
• Demonstrate that the approach is lightweight and requires no model architecture changes.

Proposed method

• Perform DDPM inversion on the input image to obtain inverted latents and noise maps.
• Encode the target prompt and SEGA concepts to obtain conditioning vectors.
• Run the denoising loop from T to 1 using the DDPM update with epsilon_theta informed by semantic guidance.
• Update latents x_{t-1} using the pre-computed noise maps Z_t within the SEGA-guided diffusion process.
• Decode the final latent x_0 to produce the edited image.
• Compare two editing workflows: pure inversion with SEGA edits and joint inversion+target-prompt editing, highlighting flexibility and robustness.

Figure 1: LEDITS- DDPM inversion with semantic guidance for real image editing. Real images edited purely with DDPM inversion and with both DDPM inversion and semantic guidance (LEDITS). In this combined approach we first apply DDPM Inversion on the input image, and then edit by performing the rever

Experimental results

Research questions

• RQ1 Can LEDITSEdit real images faithfully while allowing substantial or subtle edits?
• RQ2 Does combining DDPM inversion with SEGA preserve fidelity to the original image while enabling target-guided changes?
• RQ3 How does LEDITS compare to pure inversion or Prompt-to-Prompt in terms of flexibility and control?
• RQ4 Do SEGA guidance vectors retain robustness and monotonicity within the LEDITS framework?

Key findings

• LEDITS yields qualitative edits competitive with state-of-the-art methods without architectural changes.
• The approach provides flexible control by combining DDPM inversion with SEGA semantics.
• SEGA guidance in LEDITS preserves robustness and monotonicity properties.
• The method supports two editing workflows, enabling diversity and versatility beyond pure inversion or pure SEGA edits.
• The integration remains lightweight and maintains the strengths of both constituent techniques.

Figure 2: LEDITS overview. Top: inversion of the input image. We first apply DDPM inversion on the original image to obtain the inverted latents and corresponding noise maps. Bottom: We use the inverted latents to drive the reverse diffusion process with semantic guidance. In each denoising step we