Skip to main content
QUICK REVIEW

[论文解读] Inpaint Anything: Segment Anything Meets Image Inpainting

Changyuan Yu, Runseng Feng|arXiv (Cornell University)|Apr 13, 2023
Generative Adversarial Networks and Image SynthesisComputer Science被引用 59
一句话总结

本文提出 Inpaint Anything (IA),一个无掩码的修复管线,将 SAM、最先进的修复模型以及 AIGC 模型结合,通过简单点击和文本提示来移除、填充或替换内容。

ABSTRACT

Modern image inpainting systems, despite the significant progress, often struggle with mask selection and holes filling. Based on Segment-Anything Model (SAM), we make the first attempt to the mask-free image inpainting and propose a new paradigm of ``clicking and filling'', which is named as Inpaint Anything (IA). The core idea behind IA is to combine the strengths of different models in order to build a very powerful and user-friendly pipeline for solving inpainting-related problems. IA supports three main features: (i) Remove Anything: users could click on an object and IA will remove it and smooth the ``hole'' with the context; (ii) Fill Anything: after certain objects removal, users could provide text-based prompts to IA, and then it will fill the hole with the corresponding generative content via driving AIGC models like Stable Diffusion; (iii) Replace Anything: with IA, users have another option to retain the click-selected object and replace the remaining background with the newly generated scenes. We are also very willing to help everyone share and promote new projects based on our Inpaint Anything (IA). Our codes are available at https://github.com/geekyutao/Inpaint-Anything.

研究动机与目标

  • Motivate a mask-free inpainting paradigm that leverages segmentation and generative models.
  • Propose a user-friendly workflow for object removal, content filling, and background replacement.
  • Demonstrate how combining foundation models improves flexibility and accessibility for inpainting tasks.

提出的方法

  • Leverage Segment Anything (SAM) for accurate object masks from simple clicks.
  • Use a state-of-the-art inpainting model (e.g., LaMa) with refined masks for removal.
  • Incorporate AIGC models (e.g., Stable Diffusion) to generate content for filling or background replacement via text prompts.
  • Provide three workflows: Remove Anything, Fill Anything, and Replace Anything.
  • Apply mask refinement steps (e.g., dilation) and fidelity-preserving resizing techniques to improve results.

实验结果

研究问题

  • RQ1Can SAM-generated masks be effectively used for mask-free inpainting workflows with minimal user input?
  • RQ2How can object removal, content generation, and background replacement be unified into a single IA pipeline?
  • RQ3What are practical considerations (mask refinement, resolution handling, prompts) to achieve high-quality inpainted outputs across diverse images?

主要发现

  • IA enables three functional modes—Remove Anything, Fill Anything, and Replace Anything—using a click-plus-prompt interface.
  • The pipeline combines SAM, LaMa, and Stable Diffusion to produce high-quality inpainting results across diverse content and resolutions.
  • Mask refinement (dilation) and fidelity-preserving resizing improve alignment and detail in inpainted regions.
  • Experiments on COCO, LaMa test set, and mobile photos show IA is general and robust for varied aspect ratios and image sizes (up to 2K).
  • The approach demonstrates effective mask-free inpainting by leveraging existing foundation models in a compositional workflow.

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。