[Paper Review] LDEdit: Towards Generalized Text Guided Image Manipulation via Latent Diffusion Models

Paramanand Chandramouli, Kanchana Vaishnavi Gandikota|arXiv (Cornell University)|Oct 5, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

LDEdit proposes a zero-shot, optimization-free method for generalized text-guided image manipulation using Latent Diffusion Models (LDMs). By leveraging deterministic DDIM sampling in the latent space and conditioning reverse diffusion on target text prompts, it enables fast, flexible editing of local and global attributes—such as facial features, colors, and artistic styles—without fine-tuning, outperforming baselines in speed and user preference.

ABSTRACT

Research in vision-language models has seen rapid developments off-late, enabling natural language-based interfaces for image generation and manipulation. Many existing text guided manipulation techniques are restricted to specific classes of images, and often require fine-tuning to transfer to a different style or domain. Nevertheless, generic image manipulation using a single model with flexible text inputs is highly desirable. Recent work addresses this task by guiding generative models trained on the generic image datasets using pretrained vision-language encoders. While promising, this approach requires expensive optimization for each input. In this work, we propose an optimization-free method for the task of generic image manipulation from text prompts. Our approach exploits recent Latent Diffusion Models (LDM) for text to image generation to achieve zero-shot text guided manipulation. We employ a deterministic forward diffusion in a lower dimensional latent space, and the desired manipulation is achieved by simply providing the target text to condition the reverse diffusion process. We refer to our approach as LDEdit. We demonstrate the applicability of our method on semantic image manipulation and artistic style transfer. Our method can accomplish image manipulation on diverse domains and enables editing multiple attributes in a straightforward fashion. Extensive experiments demonstrate the benefit of our approach over competing baselines.

Motivation & Objective

• To develop a fast and flexible method for open-domain image manipulation using arbitrary text prompts.
• To overcome limitations of existing methods that require fine-tuning or are restricted to specific image classes or manipulation tasks.
• To enable zero-shot editing of multiple attributes and artistic styles using a single pretrained model.
• To achieve high inference speed and scalability while preserving content consistency and identity.
• To provide a unified framework for diverse editing tasks, including local edits, global style transfer, and stroke-to-image translation.

Proposed method

• Leverages a pretrained Latent Diffusion Model (LDM) for text-to-image generation in a lower-dimensional latent space.
• Employs deterministic forward DDIM sampling to encode the source image into a shared latent representation.
• Conditions the reverse diffusion process on both the shared latent code and the target text prompt to generate edited images.
• Uses controlled stochasticity to trade-off fidelity to the original image against diversity in the output.
• Avoids iterative optimization or fine-tuning by directly using the pretrained LDM with text conditioning.
• Applies the same model architecture and inference pipeline across diverse editing tasks, including attribute editing and style transfer.

Experimental results

Research questions

• RQ1 Can a single, pretrained LDM be used for zero-shot text-guided image manipulation without any optimization or fine-tuning?
• RQ2 How well does the method generalize across diverse editing tasks, including local attribute changes and global style transfer?
• RQ3 What is the inference speed and memory efficiency of the proposed method compared to optimization-based baselines?
• RQ4 How does user preference for LDEdit compare to existing methods like VQGAN+CLIP and DiffusionCLIP?
• RQ5 To what extent can the method preserve identity and content structure while modifying semantic attributes?

Key findings

• LDEdit achieved a user preference of 83.87% over VQGAN+CLIP in human evaluation, indicating strong perceptual quality and edit accuracy.
• User preference for LDEdit was 49.15% over DiffusionCLIP, suggesting competitive performance despite the latter's fine-tuning.
• LDEdit achieved inference times of 2.02s ± 5.58ms for a single image, significantly faster than VQGAN+CLIP (4–6 minutes) and comparable to DiffusionCLIP (4.01s–11.54s).
• LDEdit required only 8.8GB GPU memory for a single image, less than VQGAN+CLIP (10.4GB) and significantly less than DiffusionCLIP (5.4GB) for comparable performance.
• The method successfully performed simultaneous edits on multiple attributes (e.g., makeup, expression, wrinkles) while preserving identity in face images.
• LDEdit enabled mask-free local edits, such as adding a flower on a hat or glasses, using only text prompts.