[Paper Review] RMP-SAM: Towards Real-Time Multi-Purpose Segment Anything

Shilin Xu, Haobo Yuan+10 more|arXiv (Cornell University)|Jan 18, 2024

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes RAP-SAM, the first real-time multi-purpose segmentation model capable of performing interactive, panoptic, and video instance segmentation with a single unified architecture. It achieves state-of-the-art speed-accuracy trade-offs through a lightweight encoder, a unified decoder with dual adapters, and efficient pooling-based dynamic convolution, outperforming prior methods on COCO and YouTube-VIS benchmarks while running in real time.

ABSTRACT

Recent segmentation methods, which adopt large-scale data training and transformer architecture, aim to create one foundation model that can perform multiple tasks. However, most of these methods rely on heavy encoder and decoder frameworks, hindering their performance in real-time scenarios. To explore real-time segmentation, recent advancements primarily focus on semantic segmentation within specific environments, such as autonomous driving. However, they often overlook the generalization ability of these models across diverse scenarios. Therefore, to fill this gap, this work explores a novel real-time segmentation setting called real-time multi-purpose segmentation. It contains three fundamental sub-tasks: interactive segmentation, panoptic segmentation, and video instance segmentation. Unlike previous methods, which use a specific design for each task, we aim to use only a single end-to-end model to accomplish all these tasks in real-time. To meet real-time requirements and balance multi-task learning, we present a novel dynamic convolution-based method, Real-Time Multi-Purpose SAM (RMP-SAM). It contains an efficient encoder and an efficient decoupled adapter to perform prompt-driven decoding. Moreover, we further explore different training strategies and one new adapter design to boost co-training performance further. We benchmark several strong baselines by extending existing works to support our multi-purpose segmentation. Extensive experiments demonstrate that RMP-SAM is effective and generalizes well on proposed benchmarks and other specific semantic tasks. Our implementation of RMP-SAM achieves the optimal balance between accuracy and speed for these tasks.Our code and model are available at https://github.com/xushilin1/RAP-SAM/.

Motivation & Objective

• To address the lack of real-time, multi-purpose segmentation models that unify image, video, and interactive segmentation tasks.
• To develop a single model that achieves high performance across interactive segmentation, panoptic segmentation, and video instance segmentation under strict inference speed constraints.
• To explore efficient architectures and training strategies that balance performance across diverse segmentation tasks without cascaded or heavy components.
• To enable practical deployment of foundation models on edge devices by optimizing for speed and model efficiency.
• To establish a new benchmark for real-time all-purpose segmentation using COCO and YouTube-VIS datasets with joint co-training.

Proposed method

• Proposes a unified decoder with shared queries for all three tasks—interactive, panoptic, and video segmentation—using a single forward pass.
• Employs a lightweight feature extractor and pooling-based dynamic convolution to replace computationally expensive per-pixel cross-attention for faster inference.
• Introduces a dual adapter design to balance knowledge transfer between interactive and semantic-level segmentation tasks, improving co-training performance.
• Uses a single pyramid feature map for all tasks, avoiding cascaded decoder designs that increase latency.
• Applies joint co-training on COCO and YouTube-VIS datasets with identical hyperparameters to enable multi-task generalization.
• Adopts large-scale jittering (LSJ) data augmentation and a linear warmup + cosine decay learning rate schedule for stable training.

Experimental results

Research questions

• RQ1 Can a single real-time model achieve high performance across interactive, panoptic, and video instance segmentation tasks?
• RQ2 How can efficient architecture design balance accuracy and speed in multi-purpose segmentation?
• RQ3 What training strategies and adapter mechanisms best enable co-training between interactive and semantic-level segmentation?
• RQ4 Can pooling-based dynamic convolution replace cross-attention to maintain accuracy while improving inference speed?
• RQ5 How does the unified decoder with shared queries generalize across diverse segmentation tasks under real-time constraints?

Key findings

• RAP-SAM achieves the best speed-accuracy trade-off among all compared methods on the proposed real-time all-purpose segmentation benchmark.
• The model reaches 58.1 mIoU on interactive segmentation and 43.0 PQ on panoptic segmentation using ResNet-50, outperforming Mask2Former with adapter by 1.1 mIoU and only losing 0.3 PQ.
• RAP-SAM demonstrates superior edge detection and consistency in video tracking compared to Mask2Former on YouTube-VIS 2019, especially in complex and crowded scenes.
• The model maintains high performance on both image and video segmentation tasks, with smoother and more accurate masks than K-Net on COCO panoptic segmentation.
• Failure cases reveal limitations in handling high object overlap and dense instances, indicating room for improvement in instance kernel allocation.
• The dual adapter design generalizes well, improving performance across multiple models, including Mask2Former, suggesting broader applicability beyond RAP-SAM.