[Paper Review] Magicremover: Tuning-free Text-guided Image inpainting with Diffusion Models

Siyuan Yang, Lu Zhang+4 more|arXiv (Cornell University)|Oct 4, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

MagicRemover is a tuning-free, text-guided image inpainting method that leverages pretrained diffusion models to remove objects and restore backgrounds using only textual descriptions, without requiring manual masks or model fine-tuning. It employs attention-based guidance and classifier optimization to enhance denoising stability and object removal fidelity, achieving state-of-the-art performance in both quantitative metrics and user studies.

ABSTRACT

Image inpainting aims to fill in the missing pixels with visually coherent and semantically plausible content. Despite the great progress brought from deep generative models, this task still suffers from i. the difficulties in large-scale realistic data collection and costly model training; and ii. the intrinsic limitations in the traditionally user-defined binary masks on objects with unclear boundaries or transparent texture. In this paper, we propose MagicRemover, a tuning-free method that leverages the powerful diffusion models for text-guided image inpainting. We introduce an attention guidance strategy to constrain the sampling process of diffusion models, enabling the erasing of instructed areas and the restoration of occluded content. We further propose a classifier optimization algorithm to facilitate the denoising stability within less sampling steps. Extensive comparisons are conducted among our MagicRemover and state-of-the-art methods including quantitative evaluation and user study, demonstrating the significant improvement of MagicRemover on high-quality image inpainting. We will release our code at https://github.com/exisas/Magicremover.

Motivation & Objective

• To address the limitations of mask-based image inpainting, especially for objects with soft boundaries or transparency, where manual masks are impractical.
• To eliminate the need for costly data collection and model fine-tuning in image inpainting by leveraging pretrained diffusion models in a zero-shot manner.
• To enable flexible, user-friendly object removal through natural language instructions rather than binary masks.
• To improve the consistency and realism of inpainted regions by stabilizing the denoising process in diffusion sampling.

Proposed method

• The method projects input images and text prompts into the latent space of a pretrained text-to-image diffusion model to compute cross-attention and self-attention maps.
• It introduces an attention-based guidance strategy that modifies attention maps to suppress features corresponding to the target object and enhance background generation.
• A classifier optimization algorithm is applied during denoising to reduce the number of sampling steps while improving object removal accuracy and stability.
• The approach uses a reverse diffusion process to reconstruct the latent representation of the input image, enabling precise localization of the target object via attention mechanisms.
• The method avoids any additional training or finetuning, relying solely on the internal representations of the pretrained diffusion model.
• It enables soft object removal by leveraging attention maps to guide the denoising process toward erasing the target region and restoring the background contextually.

Experimental results

Research questions

• RQ1 Can a pretrained diffusion model be used for zero-shot, text-guided image inpainting without any fine-tuning or auxiliary training?
• RQ2 How can attention maps in a diffusion model be manipulated to guide object removal and background restoration effectively?
• RQ3 Can classifier optimization improve denoising stability and reduce sampling steps while maintaining high-quality inpainting results?
• RQ4 How does the method perform on objects with ambiguous or soft boundaries compared to traditional mask-based approaches?
• RQ5 What is the impact of attention guidance components on the fidelity and consistency of the inpainted output?

Key findings

• MagicRemover achieves a lower Fréchet Inception Distance (FID) than SD-Inpaint and LaMa on the COCO 2017 validation set, indicating improved image quality and distribution similarity.
• Compared to Inst-Inpaint, MagicRemover achieves better FID by avoiding residual shadows and improving object recognition accuracy.
• User studies with 30 participants show that MagicRemover received the majority of votes in preference rankings, confirming superior perceptual quality and realism.
• Abllation studies confirm that self-attention guidance preserves original image similarity, while relaxing constraints and reweight perturbation enhance background consistency.
• Classifier optimization significantly improves object removal performance even at low sampling steps (e.g., 50 steps), reducing residual object information and enhancing editing capability.
• The method successfully removes semi-transparent objects like bubbles and glass, which are challenging for binary mask-based methods due to unclear boundaries.