[论文解读] LEDITS: Real Image Editing with DDPM Inversion and Semantic Guidance

Linoy Tsaban, Apolinário Passos|arXiv (Cornell University)|Jul 2, 2023

Generative Adversarial Networks and Image Synthesis参考文献 16被引用 8

一句话总结

LEDITS 将 DDPM 反演与 SEGA 语义引导结合，在不改变模型架构的情况下以轻量、灵活的方式对真实图像进行编辑。

ABSTRACT

Recent large-scale text-guided diffusion models provide powerful image-generation capabilities. Currently, a significant effort is given to enable the modification of these images using text only as means to offer intuitive and versatile editing. However, editing proves to be difficult for these generative models due to the inherent nature of editing techniques, which involves preserving certain content from the original image. Conversely, in text-based models, even minor modifications to the text prompt frequently result in an entirely distinct result, making attaining one-shot generation that accurately corresponds to the users intent exceedingly challenging. In addition, to edit a real image using these state-of-the-art tools, one must first invert the image into the pre-trained models domain - adding another factor affecting the edit quality, as well as latency. In this exploratory report, we propose LEDITS - a combined lightweight approach for real-image editing, incorporating the Edit Friendly DDPM inversion technique with Semantic Guidance, thus extending Semantic Guidance to real image editing, while harnessing the editing capabilities of DDPM inversion as well. This approach achieves versatile edits, both subtle and extensive as well as alterations in composition and style, while requiring no optimization nor extensions to the architecture.

研究动机与目标

• 以文本引导的扩散模型激发真实图像编辑并解决编辑挑战。
• 提出将 DDPM 反演与 SEGA 的轻量级集成，以实现对真实图像的语义引导编辑。
• 证明将 DDPM 反演与 SEGA 结合能够实现多样化的编辑，同时保持保真度与语义控制。
• 展示该方法的轻量特性且无需改变模型架构。

提出的方法

• 对输入图像执行 DDPM 反演，获得反演潜变量与噪声映射。
• 对目标提示与 SEGA 概念进行编码，获得条件向量。
• 使用带有由语义引导信息影响的 epsilon_theta 的 DDPM 更新，从 T 逐步到 1 运行去噪循环。
• 在 SEGA 引导的扩散过程内，使用预计算的噪声映射 Z_t 更新潜变量 x_{t-1}。
• 对最终潜变量 x_0 进行解码，产生编辑后的图像。
• 比较两种编辑工作流：纯反演结合 SEGA 编辑，以及联合反演+目标提示编辑，突出灵活性与鲁棒性。

Figure 1: LEDITS- DDPM inversion with semantic guidance for real image editing. Real images edited purely with DDPM inversion and with both DDPM inversion and semantic guidance (LEDITS). In this combined approach we first apply DDPM Inversion on the input image, and then edit by performing the rever

实验结果

研究问题

• RQ1 LEDITS 是否在保持忠实原图的前提下实现真实图像的显著或细微编辑？
• RQ2 将 DDPM 反演与 SEGA 结合，是否在实现目标引导变化的同时保持对原图的保真度？
• RQ3 与纯反演或 Prompt-to-Prompt 相比，LEDITS 在灵活性与控制方面有何差异？
• RQ4 在 LEDITS 框架下，SEGA 引导向量是否保持鲁棒性与单调性？

主要发现

• LEDITS 在不改变架构的情况下，获得了与最先进方法相当的定性编辑结果。
• 该方法通过将 DDPM 反演与 SEGA 语义相结合，提供了灵活的控制能力。
• LEDITS 中的 SEGA 引导保持了鲁棒性与单调性属性。
• 该方法支持两种编辑工作流，超越纯反演或纯 SEGA 编辑，呈现多样性与通用性。
• 该集成保持轻量化，保留了两种组成技术的优点。

Figure 2: LEDITS overview. Top: inversion of the input image. We first apply DDPM inversion on the original image to obtain the inverted latents and corresponding noise maps. Bottom: We use the inverted latents to drive the reverse diffusion process with semantic guidance. In each denoising step we