[Paper Review] Segment Anything

Alexander M. Kirillov, Eric Mintun+10 more|arXiv (Cornell University)|Apr 5, 2023

Advanced Neural Network Applications519 citations

TL;DR

The paper presents the Segment Anything Model (SAM), a promptable segmentation model trained on SA-1B, a 1B-mask, 11M-image dataset, enabling zero-shot segmentation across tasks via prompting and data-annotation in a model-in-the-loop data engine.

ABSTRACT

We introduce the Segment Anything (SA) project: a new task, model, and dataset for image segmentation. Using our efficient model in a data collection loop, we built the largest segmentation dataset to date (by far), with over 1 billion masks on 11M licensed and privacy respecting images. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks. We evaluate its capabilities on numerous tasks and find that its zero-shot performance is impressive -- often competitive with or even superior to prior fully supervised results. We are releasing the Segment Anything Model (SAM) and corresponding dataset (SA-1B) of 1B masks and 11M images at https://segment-anything.com to foster research into foundation models for computer vision.

Motivation & Objective

• Define a promptable segmentation task to enable zero-shot generalization across segmentation tasks.
• Develop a lightweight, real-time SAM architecture that can be prompted to generate multiple valid masks.
• Create a scalable data engine to build SA-1B, the largest segmentation dataset to date, with high-quality automatic masks.
• Evaluate SAM across diverse downstream tasks and datasets to demonstrate zero-shot transfer capabilities.

Proposed method

• Propose a promptable segmentation task where a valid mask is produced for any given prompt.
• Design SAM with three components: image encoder (ViT-based), prompt encoder (points, boxes, text, or masks), and a fast mask decoder.
• Make SAM ambiguity-aware to output multiple masks per prompt with confidence scores.
• Build a three-stage data engine (assisted-manual, semi-automatic, fully automatic) to collect SA-1B masks with model-in-the-loop labeling.
• Assemble SA-1B by fully automatically generating 1.1B masks for 11M images.
• Train SAM on SA-1B and enable real-time prompting (~50 ms in a browser) using amortized image embeddings.

Experimental results

Research questions

• RQ1 What task enables zero-shot generalization in segmentation?
• RQ2 What model architecture supports promptable segmentation with real-time mask generation?
• RQ3 What data can power a promptable segmentation model and how can it be collected at scale?

Key findings

• SAM achieves zero-shot transfer to edge detection, object proposal generation, instance segmentation, and text-to-mask prompts across 23 datasets.
• SAM can produce high-quality masks from a single foreground point, often close to ground-truth, and with a human-rated quality higher than baselines.
• SAM outputs masks in approximately 50 ms given an image embedding, enabling interactive prompting.
• SA-1B contains over 1B masks from 11M licensed images, generated automatically in the final stage of the data engine, and is larger and more diverse than prior segmentation datasets.
• Mask quality in SA-1B is high, with 94% IoU > 90% for 500 image samples when paired with professional edits; 97% IoU > 75%.
• The dataset shows broad geographic and income representation improvements over prior datasets, with caveats on underrepresentation in Africa and low-income regions.