[Paper Review] Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks

Tianhe Ren, Shilong Liu+15 more|arXiv (Cornell University)|Jan 25, 2024

Multimodal Machine Learning Applications88 citations

TL;DR

The paper introduces Segment Anything (SAM), a promptable segmentation model trained on SA-1B (1B masks over 11M images) that achieves strong zero-shot performance and supports interactive, real-time mask generation across diverse tasks.

ABSTRACT

We introduce Grounded SAM, which uses Grounding DINO as an open-set object detector to combine with the segment anything model (SAM). This integration enables the detection and segmentation of any regions based on arbitrary text inputs and opens a door to connecting various vision models. As shown in Fig.1, a wide range of vision tasks can be achieved by using the versatile Grounded SAM pipeline. For example, an automatic annotation pipeline based solely on input images can be realized by incorporating models such as BLIP and Recognize Anything. Additionally, incorporating Stable-Diffusion allows for controllable image editing, while the integration of OSX facilitates promptable 3D human motion analysis. Grounded SAM also shows superior performance on open-vocabulary benchmarks, achieving 48.7 mean AP on SegInW (Segmentation in the wild) zero-shot benchmark with the combination of Grounding DINO-Base and SAM-Huge models.

Motivation & Objective

• Define a promptable segmentation task to enable zero-shot generalization across segmentation tasks.
• Develop a lightweight yet flexible model (SAM) that supports various prompts and real-time mask generation.
• Create a data engine to automatically build a massive, diverse segmentation dataset (SA-1B).
• Assess SAM's zero-shot transfer capabilities across multiple downstream tasks and distributions.
• Address responsible AI considerations and biases in data and model performance.

Proposed method

• Propose the promptable segmentation task where a valid mask is returned for any given prompt, enabling pre-training and downstream use via prompting.
• Design SAM with three components: a pre-trained image encoder, a flexible prompt encoder, and a fast mask decoder.
• Make SAM ambiguity-aware by allowing multiple masks per prompt with associated confidence scores.
• Train SAM with a mix of sparse and dense prompts using a loss that combines focal and dice losses, plus rounds of simulated prompts to reflect interactive use.
• Build a data engine with assisted-manual, semi-automatic, and fully automatic stages to collect masks in a model-in-the-loop fashion.
• Automatically generate SA-1B by applying the final, ambiguity-aware SAM to a 32x32 grid of prompts across 11M images, followed by mask refinement steps.

Experimental results

Research questions

• RQ1 What task will enable zero-shot generalization in segmentation?
• RQ2 What model architecture supports promptable segmentation with real-time performance and ambiguity handling?
• RQ3 What data scale and diversity are needed to train a robust promptable segmentation model?
• RQ4 Can a promptable segmentation model transfer effectively to downstream tasks via prompting?
• RQ5 How does SAM perform across diverse datasets and distributions in zero-shot settings?

Key findings

• SAM achieves high-quality masks from a single foreground point, often close to ground-truth performance.
• SAM demonstrates strong zero-shot transfer across 23 segmentation datasets, often outperforming or matching specialized baselines.
• The SA-1B dataset contains over 1.1B masks on 11M images, significantly exceeding prior datasets in both scale and diversity.
• The data engine and fully automatic stage enable scalable mask generation without compromising quality (high IoU with professional annotations in a sample).
• Ambiguity-aware prompting yields multiple valid masks with confidence scores, improving handling of ambiguous prompts.