[논문 리뷰] Inpaint Anything: Segment Anything Meets Image Inpainting

Changyuan Yu, Runseng Feng외 5명|arXiv (Cornell University)|2023. 04. 13.

Generative Adversarial Networks and Image Synthesis인용 수 59

한 줄 요약

The paper introduces Inpaint Anything (IA), a mask-free inpainting pipeline that combines SAM, a state-of-the-art inpainting model, and AIGC models to remove, fill, or replace content via simple clicks and text prompts.

ABSTRACT

Modern image inpainting systems, despite the significant progress, often struggle with mask selection and holes filling. Based on Segment-Anything Model (SAM), we make the first attempt to the mask-free image inpainting and propose a new paradigm of ``clicking and filling'', which is named as Inpaint Anything (IA). The core idea behind IA is to combine the strengths of different models in order to build a very powerful and user-friendly pipeline for solving inpainting-related problems. IA supports three main features: (i) Remove Anything: users could click on an object and IA will remove it and smooth the ``hole'' with the context; (ii) Fill Anything: after certain objects removal, users could provide text-based prompts to IA, and then it will fill the hole with the corresponding generative content via driving AIGC models like Stable Diffusion; (iii) Replace Anything: with IA, users have another option to retain the click-selected object and replace the remaining background with the newly generated scenes. We are also very willing to help everyone share and promote new projects based on our Inpaint Anything (IA). Our codes are available at https://github.com/geekyutao/Inpaint-Anything.

연구 동기 및 목표

• Motivate a mask-free inpainting paradigm that leverages segmentation and generative models.
• Propose a user-friendly workflow for object removal, content filling, and background replacement.
• Demonstrate how combining foundation models improves flexibility and accessibility for inpainting tasks.

제안 방법

• Leverage Segment Anything (SAM) for accurate object masks from simple clicks.
• Use a state-of-the-art inpainting model (e.g., LaMa) with refined masks for removal.
• Incorporate AIGC models (e.g., Stable Diffusion) to generate content for filling or background replacement via text prompts.
• Provide three workflows: Remove Anything, Fill Anything, and Replace Anything.
• Apply mask refinement steps (e.g., dilation) and fidelity-preserving resizing techniques to improve results.

실험 결과

연구 질문

• RQ1 Can SAM-generated masks be effectively used for mask-free inpainting workflows with minimal user input?
• RQ2 How can object removal, content generation, and background replacement be unified into a single IA pipeline?
• RQ3 What are practical considerations (mask refinement, resolution handling, prompts) to achieve high-quality inpainted outputs across diverse images?

주요 결과

• IA enables three functional modes—Remove Anything, Fill Anything, and Replace Anything—using a click-plus-prompt interface.
• The pipeline combines SAM, LaMa, and Stable Diffusion to produce high-quality inpainting results across diverse content and resolutions.
• Mask refinement (dilation) and fidelity-preserving resizing improve alignment and detail in inpainted regions.
• Experiments on COCO, LaMa test set, and mobile photos show IA is general and robust for varied aspect ratios and image sizes (up to 2K).
• The approach demonstrates effective mask-free inpainting by leveraging existing foundation models in a compositional workflow.