[论文解读] Image Segmentation in Foundation Model Era: A Survey

Tianfei Zhou, Xia, Wang+6 more|arXiv (Cornell University)|Aug 23, 2024

Brain Tumor Detection and Classification被引用 9

一句话总结

对基础模型如何实现通用且可提示的图像分割的综合综述，强调来自 CLIP、扩散模型、DINO 和 SAM 的新兴分割知识，并概述开放挑战与未来方向。

ABSTRACT

Image segmentation is a long-standing challenge in computer vision, studied continuously over several decades, as evidenced by seminal algorithms such as N-Cut, FCN, and MaskFormer. With the advent of foundation models (FMs), contemporary segmentation methodologies have embarked on a new epoch by either adapting FMs (e.g., CLIP, Stable Diffusion, DINO) for image segmentation or developing dedicated segmentation foundation models (e.g., SAM). These approaches not only deliver superior segmentation performance, but also herald newfound segmentation capabilities previously unseen in deep learning context. However, current research in image segmentation lacks a detailed analysis of distinct characteristics, challenges, and solutions associated with these advancements. This survey seeks to fill this gap by providing a thorough review of cutting-edge research centered around FM-driven image segmentation. We investigate two basic lines of research -- generic image segmentation (i.e., semantic segmentation, instance segmentation, panoptic segmentation), and promptable image segmentation (i.e., interactive segmentation, referring segmentation, few-shot segmentation) -- by delineating their respective task settings, background concepts, and key challenges. Furthermore, we provide insights into the emergence of segmentation knowledge from FMs like CLIP, Stable Diffusion, and DINO. An exhaustive overview of over 300 segmentation approaches is provided to encapsulate the breadth of current research efforts. Subsequently, we engage in a discussion of open issues and potential avenues for future research. We envisage that this fresh, comprehensive, and systematic survey catalyzes the evolution of advanced image segmentation systems. A public website is created to continuously track developments in this fast advancing field: \url{https://github.com/stanley-313/ImageSegFM-Survey}.

研究动机与目标

• 解释基础模型如何将图像分割转换为通用、可提示的任务。
• 回顾 CLIP、扩散模型、DINO、SAM 及其他用于分割知识出现的基础模型。
• 对 GIS 和可提示分割在开放词汇与封闭词汇设置下进行分类和分析。
• 讨论训练无关分割与提示策略，以在不同任务中统一多样化的提示。
• 识别开放问题和潜在方向，以指导未来基于基础模型的分割研究。

提出的方法

• 给出分割作为从输入到掩模与词汇表的映射 f 的统一数学表述。
• 建立将通用图像分割与可提示图像分割区分的分类法。
• 调查基础模型及其知识出现机制，涉及 CLIP、扩散模型、DINO 和 SAM。
• 穷尽性地评审将分割能力从基础模型迁移或提取的方法。
• 讨论训练无关分割与基于提示的界面，实现零-shot 和少-shot 能力。

Figure 1: The taxonomy based on image segmentation tasks and foundation models. Typical solutions are shown for each category.

实验结果

研究问题

• RQ1 基础模型如何在 GIS 和 PIS 任务中实现通用、可提示的分割？
• RQ2 哪些机制使分割知识能够从 CLIP、DINO、扩散模型和 SAM 中出现？
• RQ3 在基础模型下实现开放词汇和训练无关分割的有效策略有哪些？
• RQ4 基于基础模型的图像分割有哪些关键开放问题和未来方向？

主要发现

• 基础模型能够成为对多种分割任务具备提示性和可适应性的分割通才。
• 分割知识可以来自 CLIP 的对齐与注意力机制、DINO、扩散模型，以及 SAM 的跨注意力。
• 训练无关和少样本提示方法使得在没有特定任务训练的情况下实现零-shot 或少-shot分割成为可能。
• 由于基础模型的能力，开放词汇分割正得到越来越多的关注，超越封闭词汇设置。
• 存在一系列基于基础模型的分割技术，覆盖语义、实例和全景任务，且具有实用的交互式和文本驱动提示。

Figure 2: MLLMs driven solutions lead to more powerful pixel reasoning and understanding capabilities, e.g . , multi-target reasoning segmentation, instance segmentation with text descriptions, referring segmentation and conversation. (Figure adapted courtesy of [ 60 ] )