[Paper Review] PFB-Diff: Progressive Feature Blending Diffusion for Text-driven Image Editing

Wenjing Huang, Shikui Tu+1 more|arXiv (Cornell University)|Jun 28, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

PFB-Diff proposes a novel, optimization-free method for text-driven image editing using progressive feature blending and attention masking in pre-trained diffusion models. By blending deep features across multi-scale levels and confining word attention to specific regions, it achieves superior image fidelity, semantic consistency, and editing accuracy without fine-tuning.

ABSTRACT

Diffusion models have showcased their remarkable capability to synthesize diverse and high-quality images, sparking interest in their application for real image editing. However, existing diffusion-based approaches for local image editing often suffer from undesired artifacts due to the pixel-level blending of the noised target images and diffusion latent variables, which lack the necessary semantics for maintaining image consistency. To address these issues, we propose PFB-Diff, a Progressive Feature Blending method for Diffusion-based image editing. Unlike previous methods, PFB-Diff seamlessly integrates text-guided generated content into the target image through multi-level feature blending. The rich semantics encoded in deep features and the progressive blending scheme from high to low levels ensure semantic coherence and high quality in edited images. Additionally, we introduce an attention masking mechanism in the cross-attention layers to confine the impact of specific words to desired regions, further improving the performance of background editing. PFB-Diff can effectively address various editing tasks, including object/background replacement and object attribute editing. Our method demonstrates its superior performance in terms of image fidelity, editing accuracy, efficiency, and faithfulness to the original image, without the need for fine-tuning or training.

Motivation & Objective

• Address image inconsistency and image-text misalignment in optimization-free diffusion-based image editing.
• Overcome limitations of pixel-level blending in noisy intermediate features, which leads to artifacts and poor semantic coherence.
• Enable precise, localized editing by confining text-guided generation to user-defined regions via attention masking.
• Maintain high efficiency and generalization by avoiding fine-tuning while leveraging pre-trained models.
• Support diverse editing tasks, including object replacement, background editing, and attribute manipulation, with minimal user input.

Proposed method

• Apply progressive feature blending by editing deep feature maps from high to low levels using multi-scale masks, ensuring semantic consistency.
• Replace pixel-level denoising blending with feature-level fusion, where generated content is seamlessly integrated via multi-level features.
• Introduce an attention masking mechanism in cross-attention layers to restrict the influence of specific words to designated regions.
• Use a frozen pre-trained text-to-image diffusion model, avoiding any fine-tuning or optimization during inference.
• Leverage the model’s generalization capacity by conditioning the denoising process on both the original image and text prompt via masked attention.
• Enable plug-and-play deployment by decoupling the PFB and attention masking modules from the base diffusion model.

Experimental results

Research questions

• RQ1 Can feature-level editing improve semantic consistency and reduce artifacts in text-driven image editing compared to pixel-level blending?
• RQ2 How effective is progressive multi-scale feature blending in preserving image structure and generating coherent edits?
• RQ3 To what extent does attention masking improve localization accuracy and reduce unintended object generation in edited regions?
• RQ4 Can a no-fine-tuning, optimization-free method achieve state-of-the-art performance in image fidelity, alignment, and efficiency?
• RQ5 How robust is the method to coarse or imperfect user-provided masks in real-world editing scenarios?

Key findings

• PFB-Diff achieves the highest CLIP score (32.3) and Local CLIP score (30.12) on background editing tasks, outperforming baseline methods.
• The method reduces unwanted object generation—such as spurious horses around the original subject—by 100% when attention masking is applied.
• User study shows PFB-Diff achieves an average ranking score of 3.7 out of 4, significantly higher than competing methods, indicating strong human preference.
• Even with coarse or mismatched masks (e.g., generating a horse in a bird-shaped region), PFB-Diff maintains high-quality and coherent results.
• The FID score of 13.4 for PFB-Diff with both PFB and attention masking is competitive despite slightly higher than baseline, indicating strong trade-off between fidelity and alignment.
• The ablation study confirms that PFB alone improves blending quality over pixel-level methods, while attention masking is critical for precise localization.