[Paper Review] Image Segmentation in Foundation Model Era: A Survey

Tianfei Zhou, Xia, Wang+6 more|arXiv (Cornell University)|Aug 23, 2024

Brain Tumor Detection and Classification9 citations

TL;DR

A comprehensive survey on how foundation models enable generic and promptable image segmentation, highlighting emergent segmentation knowledge from CLIP, diffusion models, DINO, and SAM, and outlining open challenges and future directions.

ABSTRACT

Image segmentation is a long-standing challenge in computer vision, studied continuously over several decades, as evidenced by seminal algorithms such as N-Cut, FCN, and MaskFormer. With the advent of foundation models (FMs), contemporary segmentation methodologies have embarked on a new epoch by either adapting FMs (e.g., CLIP, Stable Diffusion, DINO) for image segmentation or developing dedicated segmentation foundation models (e.g., SAM). These approaches not only deliver superior segmentation performance, but also herald newfound segmentation capabilities previously unseen in deep learning context. However, current research in image segmentation lacks a detailed analysis of distinct characteristics, challenges, and solutions associated with these advancements. This survey seeks to fill this gap by providing a thorough review of cutting-edge research centered around FM-driven image segmentation. We investigate two basic lines of research -- generic image segmentation (i.e., semantic segmentation, instance segmentation, panoptic segmentation), and promptable image segmentation (i.e., interactive segmentation, referring segmentation, few-shot segmentation) -- by delineating their respective task settings, background concepts, and key challenges. Furthermore, we provide insights into the emergence of segmentation knowledge from FMs like CLIP, Stable Diffusion, and DINO. An exhaustive overview of over 300 segmentation approaches is provided to encapsulate the breadth of current research efforts. Subsequently, we engage in a discussion of open issues and potential avenues for future research. We envisage that this fresh, comprehensive, and systematic survey catalyzes the evolution of advanced image segmentation systems. A public website is created to continuously track developments in this fast advancing field: \url{https://github.com/stanley-313/ImageSegFM-Survey}.

Motivation & Objective

• Explain how foundation models transform image segmentation into generic, promptable tasks.
• Review CLIP, diffusion models, DINO, SAM, and other FMs for segmentation knowledge emergence.
• Categorize and analyze GIS and promptable segmentation approaches under open- and closed-vocabulary settings.
• Discuss training-free segmentation and prompting strategies to unify diverse prompts across tasks.
• Identify open issues and potential directions to guide future FM-based segmentation research.

Proposed method

• Provide a unified mathematical formulation of segmentation as a mapping f from inputs to masks and vocabularies.
• Develop a taxonomy separating generic image segmentation and promptable image segmentation.
• Survey FMs and knowledge emergence mechanisms from CLIP, diffusion models, DINO, and SAM.
• Exhaustively review methods that transfer or extract segmentation capabilities from FMs.
• Discuss training-free segmentation and prompt-based interfaces enabling zero-shot and few-shot capabilities.

Figure 1: The taxonomy based on image segmentation tasks and foundation models. Typical solutions are shown for each category.

Experimental results

Research questions

• RQ1 How do foundation models enable general-purpose, promptable segmentation across GIS and PIS tasks?
• RQ2 What mechanisms allow segmentation knowledge to emerge from CLIP, DINO, diffusion models, and SAM?
• RQ3 What are effective strategies to achieve open-vocabulary and training-free segmentation with FMs?
• RQ4 What are the key open issues and future directions for FM-based image segmentation?

Key findings

• Foundation models enable segmentation generalists that are promptable and adaptable to multiple segmentation tasks.
• Segmentation knowledge can emerge from alignment and attention mechanisms in CLIP, DINO, diffusion models, and cross-attention in SAM.
• Training-free and few-shot prompting approaches enable zero-shot or few-shot segmentation without task-specific training.
• Open-vocabulary segmentation is increasingly emphasized due to FM capabilities, expanding beyond closed-vocabulary setups.
• There is a spectrum of FM-based segmentation techniques spanning semantic, instance, and panoptic tasks, with practical interactive and text-driven prompts.

Figure 2: MLLMs driven solutions lead to more powerful pixel reasoning and understanding capabilities, e.g . , multi-target reasoning segmentation, instance segmentation with text descriptions, referring segmentation and conversation. (Figure adapted courtesy of [ 60 ] )