[Paper Review] SAM-PARSER: Fine-tuning SAM Efficiently by Parameter Space Reconstruction

Zelin Peng, Zhengqin Xu+3 more|arXiv (Cornell University)|Aug 28, 2023

Advanced Neural Network ApplicationsComputer Science3 citations

TL;DR

This paper proposes SAM-PARSER, a parameter-efficient fine-tuning method for Segment Anything Model (SAM) that reconstructs downstream task-specific parameter spaces using singular value decomposition (SVD) of SAM’s original weights, enabling fine-tuning with nearly zero additional trainable parameters. The method achieves state-of-the-art segmentation performance across natural, remote sensing, and medical image datasets while reducing trainable parameters by approximately 290× compared to existing methods.

ABSTRACT

Segment Anything Model (SAM) has received remarkable attention as it offers a powerful and versatile solution for object segmentation in images. However, fine-tuning SAM for downstream segmentation tasks under different scenarios remains a challenge, as the varied characteristics of different scenarios naturally requires diverse model parameter spaces. Most existing fine-tuning methods attempt to bridge the gaps among different scenarios by introducing a set of new parameters to modify SAM's original parameter space. Unlike these works, in this paper, we propose fine-tuning SAM efficiently by parameter space reconstruction (SAM-PARSER), which introduce nearly zero trainable parameters during fine-tuning. In SAM-PARSER, we assume that SAM's original parameter space is relatively complete, so that its bases are able to reconstruct the parameter space of a new scenario. We obtain the bases by matrix decomposition, and fine-tuning the coefficients to reconstruct the parameter space tailored to the new scenario by an optimal linear combination of the bases. Experimental results show that SAM-PARSER exhibits superior segmentation performance across various scenarios, while reducing the number of trainable parameters by $\approx 290$ times compared with current parameter-efficient fine-tuning methods.

Motivation & Objective

• To address the challenge of fine-tuning SAM for diverse downstream segmentation scenarios with minimal parameter updates.
• To reduce the number of trainable parameters in SAM fine-tuning while maintaining or improving segmentation performance.
• To explore whether SAM’s original parameter space is sufficiently complete to reconstruct new task-specific parameter spaces without introducing new parameters.
• To develop a method that avoids adding new trainable heads or adapters, instead relying on coefficient fine-tuning of decomposed bases.

Proposed method

• The method performs singular value decomposition (SVD) on the convolutional encoder weights of SAM to extract orthogonal bases and associated coefficients.
• It assumes the original parameter space is sufficiently complete to reconstruct new task-specific parameter spaces via linear combinations of these bases.
• Only the SVD coefficients are fine-tuned during adaptation, resulting in nearly zero new trainable parameters.
• The approach focuses exclusively on the convolutional layers of SAM’s encoder, not the Transformer layers, to minimize parameter overhead.
• The optimal linear combination of bases is learned by fine-tuning the coefficients to match the target scenario’s parameter space.
• The method is applied across three distinct scenarios: natural image, remote sensing, and medical image segmentation.

Experimental results

Research questions

• RQ1 Can SAM’s original parameter space be sufficiently reconstructed to adapt to new downstream segmentation tasks without introducing new parameters?
• RQ2 How does parameter space reconstruction via SVD-based coefficient fine-tuning compare to existing parameter-efficient methods like LoRA and Adapter in segmentation performance and parameter efficiency?
• RQ3 In what scenarios does the assumption of completeness in SAM’s original parameter space break down, and how can it be mitigated?
• RQ4 Does focusing fine-tuning on the convolutional encoder layers alone yield better efficiency and performance than fine-tuning the full model or Transformer components?

Key findings

• SAM-PARSER reduces the number of trainable parameters by approximately 290 times compared to LoRA, a leading parameter-efficient method.
• On the PASCAL VOC2012 dataset, SAM-PARSER achieves a mIoU of 86.7%, outperforming the baseline and LoRA.
• On the NWPU VHR-10 remote sensing dataset, SAM-PARSER improves mIoU by 1.3 percentage points over the baseline.
• On the CT Abdominal organ test set, SAM-PARSER achieves a DSC of 91.1%, surpassing LoRA’s 90.3% by 0.8 percentage points.
• In a failure case on the SSDD radar image dataset, SAM-PARSER underperforms LoRA and Adapter, indicating limitations when the new domain is too different from SAM’s pretraining distribution.
• Qualitative results show that SAM-PARSER produces more accurate and consistent segmentation masks than LoRA and Adapter across diverse scenarios.