[Paper Review] Relax Image-Specific Prompt Requirement in SAM: A Single Generic Prompt for Segmenting Camouflaged Objects

Jian Hu, Jiayi Lin+2 more|arXiv (Cornell University)|Dec 12, 2023

Visual Attention and Saliency Detection4 citations

TL;DR

This paper proposes Generalizable SAM (GenSAM), a test-time adaptation framework that eliminates the need for image-specific prompts in Segment Anything Model (SAM) for weakly supervised camouflaged object detection. By using Cross-modal Chains of Thought Prompting (CCTP) and Progressive Mask Generation (PMG), GenSAM automatically generates accurate visual prompts from a single generic text prompt, achieving performance comparable to scribble supervision without any fine-tuning or additional training.

ABSTRACT

Camouflaged object detection (COD) approaches heavily rely on pixel-level annotated datasets. Weakly-supervised COD (WSCOD) approaches use sparse annotations like scribbles or points to reduce annotation effort, but this can lead to decreased accuracy. The Segment Anything Model (SAM) shows remarkable segmentation ability with sparse prompts like points. However, manual prompt is not always feasible, as it may not be accessible in real-world application. Additionally, it only provides localization information instead of semantic one, which can intrinsically cause ambiguity in interpreting the targets. In this work, we aim to eliminate the need for manual prompt. The key idea is to employ Cross-modal Chains of Thought Prompting (CCTP) to reason visual prompts using the semantic information given by a generic text prompt. To that end, we introduce a test-time adaptation per-instance mechanism called Generalizable SAM (GenSAM) to automatically enerate and optimize visual prompts the generic task prompt for WSCOD. In particular, CCTP maps a single generic text prompt onto image-specific consensus foreground and background heatmaps using vision-language models, acquiring reliable visual prompts. Moreover, to test-time adapt the visual prompts, we further propose Progressive Mask Generation (PMG) to iteratively reweight the input image, guiding the model to focus on the targets in a coarse-to-fine manner. Crucially, all network parameters are fixed, avoiding the need for additional training. Experiments demonstrate the superiority of GenSAM. Experiments on three benchmarks demonstrate that GenSAM outperforms point supervision approaches and achieves comparable results to scribble supervision ones, solely relying on general task descriptions as prompts. our codes is in: https://lwpyh.github.io/GenSAM/.

Motivation & Objective

• Address the limitation of manual, image-specific prompts in SAM for weakly supervised camouflaged object detection (WSCOD), which is impractical in real-world scenarios.
• Reduce annotation burden by replacing sparse annotations like points or scribbles with a single generic task description.
• Eliminate ambiguity in prompt interpretation caused by lack of semantic context in spatial prompts like points.
• Enable automatic, instance-wise visual prompt generation at test time without model fine-tuning.
• Achieve high segmentation accuracy comparable to scribble supervision using only a generic text prompt.

Proposed method

• Introduce Cross-modal Chains of Thought Prompting (CCTP) to generate multiple semantic keywords from a generic text prompt using vision-language models (BLIP2 and CLIP).
• Use a novel k-k-v self-attention mechanism to fuse keywords from different chains into consensus foreground and background heatmaps, resolving ambiguity from individual chains.
• Apply Progressive Mask Generation (PMG) to iteratively reweight the input image using consensus heatmaps, guiding SAM to focus on the target in a coarse-to-fine manner.
• Fix all network parameters during inference, enabling test-time adaptation without any fine-tuning or additional training.
• Transform the final consensus heatmap into point prompts via upsampling and thresholding, ensuring compatibility with SAM’s prompt interface.
• Evaluate multiple prompt generation strategies (e.g., MaxIOUBox) to refine mask consistency across iterations.

Experimental results

Research questions

• RQ1 Can a single generic text prompt replace image-specific prompts in SAM for camouflaged object segmentation without performance degradation?
• RQ2 How can semantic information from a generic prompt be effectively mapped to image-specific visual prompts to resolve ambiguity in localization?
• RQ3 To what extent can test-time adaptation with iterative refinement improve segmentation accuracy without model fine-tuning?
• RQ4 How does the number of chains in chain-of-thought prompting affect the quality of consensus heatmaps and final segmentation performance?
• RQ5 Can the proposed method generalize to other segmentation tasks beyond camouflaged object detection?

Key findings

• GenSAM achieves mean absolute error (MAE) of 0.025 on the S-COD dataset, outperforming point-based supervision and matching scribble-based methods.
• On the CHAMELEON dataset, GenSAM achieves an Fβ score of 0.806, significantly outperforming CLIP Surgery+SAM (0.689) and SAM-P (0.712).
• The use of consensus heatmaps as visual prompts improves performance over baseline CLIP Surgery+SAM, with MAE reduced from 0.537 to 0.205 on ETIS for polyp segmentation.
• Progressive Mask Generation (PMG) shows significant performance gain within the first 6 iterations, with minimal improvement beyond that, justifying 6 iterations as optimal.
• Using k-k-v self-attention in CCTP yields better performance than v-v-v self-attention, with a 0.027 drop in Fβ when replaced.
• MaxIOUBox outperforms other box prompt generation strategies, achieving the highest Fβ and Sα scores in ablation studies.