[Paper Review] Blended Latent Diffusion

Omri Avrahami, Ohad Fried+1 more|arXiv (Cornell University)|Jun 6, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces Blended Latent Diffusion, a zero-shot, accelerated method for local text-guided image editing that operates in the latent space of a pretrained text-to-image diffusion model. By blending latents during diffusion and optimizing reconstructions, it achieves faster inference and higher precision than prior methods, especially with thin masks and sensitive image regions.

ABSTRACT

The tremendous progress in neural image generation, coupled with the emergence of seemingly omnipotent vision-language models has finally enabled text-based interfaces for creating and editing images. Handling generic images requires a diverse underlying generative model, hence the latest works utilize diffusion models, which were shown to surpass GANs in terms of diversity. One major drawback of diffusion models, however, is their relatively slow inference time. In this paper, we present an accelerated solution to the task of local text-driven editing of generic images, where the desired edits are confined to a user-provided mask. Our solution leverages a recent text-to-image Latent Diffusion Model (LDM), which speeds up diffusion by operating in a lower-dimensional latent space. We first convert the LDM into a local image editor by incorporating Blended Diffusion into it. Next we propose an optimization-based solution for the inherent inability of this LDM to accurately reconstruct images. Finally, we address the scenario of performing local edits using thin masks. We evaluate our method against the available baselines both qualitatively and quantitatively and demonstrate that in addition to being faster, our method achieves better precision than the baselines while mitigating some of their artifacts.

Motivation & Objective

• Address the lack of efficient, precise local text-driven image editing methods for generic images.
• Overcome the slow inference speed of pixel-level diffusion models in local editing tasks.
• Enable high-fidelity local edits in latent space while preserving original image details, especially in sensitive regions like faces or text.
• Solve the challenge of editing within thin masks, which are difficult to handle in low-resolution latent spaces.
• Provide a zero-shot solution using only pretrained models, avoiding the need for fine-tuning or additional training.

Proposed method

• Adapt Blended Diffusion to operate in the latent space of a Latent Diffusion Model (LDM), replacing pixel-level blending with latent-level blending to accelerate inference.
• Introduce a latent optimization technique to correct for reconstruction errors caused by VAE-based lossy encodings in LDM, improving fidelity in high-sensitivity regions.
• Propose a dilated mask shrinking strategy to enable precise editing within thin masks by gradually reducing mask size during the diffusion process.
• Leverage CLIP for zero-shot text-image alignment to rank multiple generation results, ensuring better alignment with the input prompt.
• Use a pretrained LDM and CLIP model as foundational components, avoiding any fine-tuning or retraining.
• Combine latent blending with iterative optimization to balance speed, precision, and semantic consistency during local edits.

Experimental results

Research questions

• RQ1 Can latent-space diffusion be effectively adapted for local, text-guided image editing while maintaining high speed and precision?
• RQ2 How can reconstruction artifacts in LDMs be mitigated during local edits, especially in perceptually sensitive regions?
• RQ3 What strategies enable accurate local editing when the mask is thin or spatially constrained in low-resolution latent space?
• RQ4 How does the proposed method compare in speed and quality to existing baselines like Blended Diffusion and GLIDE?
• RQ5 To what extent is the method sensitive to small changes in input prompts, masks, or image content?

Key findings

• The method achieves significantly faster inference than pixel-level baselines, with the full-batch version of Blended Diffusion serving as the relevant comparison due to CLIP ranking.
• Quantitative evaluation shows statistically significant improvements in both visual quality and text alignment (p < 0.001), outperforming Blended Diffusion, GLIDE-filtered, and PaintByWord++.
• The method produces more precise edits, especially in thin-mask scenarios, due to the proposed mask shrinking strategy.
• Latent optimization effectively reduces reconstruction errors, particularly in high-frequency or perceptually sensitive areas like faces and text.
• The method demonstrates moderate sensitivity to input changes—small prompt or mask variations lead to small, plausible output changes, indicating controllability.
• CLIP-based ranking is crucial for performance, as single-sample outputs often fail to match the prompt, and ranking improves consistency across generations.