[Paper Review] Diving into Underwater: Segment Anything Model Guided Underwater Salient Instance Segmentation and A Large-scale Dataset

Shijie Lian, Ziyi Zhang+5 more|arXiv (Cornell University)|Jun 10, 2024

Underwater Acoustics Research5 citations

TL;DR

Introduces USIS10K, a large-scale underwater salient instance segmentation dataset, and USIS-SAM, a SAM-based model with an Underwater Adaptive ViT encoder and a Salient Feature Prompter Generator for end-to-end underwater salient instance segmentation.

ABSTRACT

With the breakthrough of large models, Segment Anything Model (SAM) and its extensions have been attempted to apply in diverse tasks of computer vision. Underwater salient instance segmentation is a foundational and vital step for various underwater vision tasks, which often suffer from low segmentation accuracy due to the complex underwater circumstances and the adaptive ability of models. Moreover, the lack of large-scale datasets with pixel-level salient instance annotations has impeded the development of machine learning techniques in this field. To address these issues, we construct the first large-scale underwater salient instance segmentation dataset (USIS10K), which contains 10,632 underwater images with pixel-level annotations in 7 categories from various underwater scenes. Then, we propose an Underwater Salient Instance Segmentation architecture based on Segment Anything Model (USIS-SAM) specifically for the underwater domain. We devise an Underwater Adaptive Visual Transformer (UA-ViT) encoder to incorporate underwater domain visual prompts into the segmentation network. We further design an out-of-the-box underwater Salient Feature Prompter Generator (SFPG) to automatically generate salient prompters instead of explicitly providing foreground points or boxes as prompts in SAM. Comprehensive experimental results show that our USIS-SAM method can achieve superior performance on USIS10K datasets compared to the state-of-the-art methods. Datasets and codes are released on https://github.com/LiamLian0727/USIS10K.

Motivation & Objective

• Establish a large-scale underwater salient instance segmentation dataset with pixel-level annotations across multiple categories.
• Adapt Segment Anything Model to underwater domains to improve segmentation accuracy in challenging marine environments.
• Develop automatic saliency prompting to enable end-to-end SAM segmentation without manual prompts.
• Demonstrate the effectiveness and generalization of the proposed USIS-SAM on USIS10K and compare with state-of-the-art methods.

Proposed method

• Create USIS10K: 10,632 underwater images with pixel-level masks across 7 categories; annotations include category labels, masks, and bounding boxes.
• Propose USIS-SAM: a SAM-based architecture tailored for underwater scenes.
• Introduce UA-ViT: a pair of adapters (image attention and channel) that fine-tunes SAM encoder blocks via PEFT to inject underwater domain knowledge.
• Develop SFPG: a Salient Feature Prompt Generator that fuses multi-scale UA-ViT features to produce prompts for SAM without manual inputs.
• Incorporate multi-scale feature fusion (SFFM) and multi-scale deconvolution to align saliency cues with the SAM decoder.
• Optimize with a loss combining RPN localization, classification, regression, and segmentation losses, following Mask RCNN-inspired objectives.

Experimental results

Research questions

• RQ1 Can a large-scale underwater salient instance segmentation dataset (USIS10K) be constructed with pixel-level annotations across multiple categories?
• RQ2 Does SAM, when augmented with underwater-specific adapters and an automatic saliency prompter, achieve superior underwater SIS performance?
• RQ3 How do underwater domain adaptations (UA-ViT and SFPG) affect segmentation accuracy and robustness in underwater scenes?
• RQ4 Can the proposed USIS-SAM generalize to terrestrial SIS datasets (e.g., SIS10K) without overfitting to USIS10K?

Key findings

• USIS-SAM achieves superior performance on USIS10K compared with state-of-the-art methods in both class-agnostic and multi-class salient instance segmentation.
• Ablation shows UA-ViT improves AP by 1.6 points, and SFPG contributes meaningful gains over alternatives.
• USIS10K is the first large-scale underwater SIS dataset with 10,632 images across 7 categories and includes both class-agnostic and multi-class labels.
• On SIS10K, USIS-SAM achieves competitive generalization, indicating the prompt-learning approach can transfer to land datasets, though with slightly different AP75.
• Qualitative results demonstrate more accurate and complete salient masks in underwater scenes when using USIS-SAM.