[논문 리뷰] Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model

Zihan Zhong, Zhiqiang Tang외 3명|arXiv (Cornell University)|2024. 01. 31.

Context-Aware Activity Recognition Systems인용 수 12

한 줄 요약

Conv-LoRA는 MoE 안내를 받는 스케일별 전문가 세트를 통해 SAM의 ViT 인코더에 가벼운 컨볼루션 선행 정보를 주입하는 매개변수 효율적인 파인튜닝 방법으로, 대부분의 SAM 가중치를 고정한 채 다양한 도메인에서 시맨틱 세그멘테이션을 향상시킨다.

ABSTRACT

The Segment Anything Model (SAM) stands as a foundational framework for image segmentation. While it exhibits remarkable zero-shot generalization in typical scenarios, its advantage diminishes when applied to specialized domains like medical imagery and remote sensing. To address this limitation, this paper introduces Conv-LoRA, a simple yet effective parameter-efficient fine-tuning approach. By integrating ultra-lightweight convolutional parameters into Low-Rank Adaptation (LoRA), Conv-LoRA can inject image-related inductive biases into the plain ViT encoder, further reinforcing SAM's local prior assumption. Notably, Conv-LoRA not only preserves SAM's extensive segmentation knowledge but also revives its capacity of learning high-level image semantics, which is constrained by SAM's foreground-background segmentation pretraining. Comprehensive experimentation across diverse benchmarks spanning multiple domains underscores Conv-LoRA's superiority in adapting SAM to real-world semantic segmentation tasks.

연구 동기 및 목표

• SAM의 제로샷 성능이 취약한 도메인 특화 세그멘테이션에서 성능을 개선하려는 동기를 제시한다(의료, 원격 감지 등).
• SAM의 지식을 보존하면서 이미지 관련 로컬 프라이어를 가능하게 하는 매개변수 효율적인 파인튜닝 방법을 제안한다.
• LoRA를 가벼운 컨볼루션과 다중 스케일 특징 처리를 위한 전문가 혼합으로 확장하여 Conv-LoRA를 개발한다.
• Conv-LoRA가 자연 이미지, 농업, 원격 감지 및 의료 데이터셋 전반에서 다른 PEFT 방법보다 우수함을 시연한다.]
• method:[
• Build on LoRA by inserting a bottleneck around the transformer weights and adding lightweight convolutions (Conv-LoRA).
• Use a mixture-of-experts (MoE) to create multiple scale-specific convolution experts and a gating mechanism that selects top-k experts dynamically during the forward pass.
• Inject local priors at appropriate feature scales by having each expert upsample, convolve, and downsample feature maps back to the ViT’s default scale.
• Remove the prompt encoder for end-to-end finetuning and add a lightweight classification branch in the mask decoder for multi-class segmentation.
• Train all methods with a small set of trainable parameters while freezing SAM’s pretrained weights; use an auxiliary loss to balance expert usage.
• Compare Conv-LoRA against baselines including decoder-only fine-tuning, BitFit, Adapter, SAM-Adapter, VPT, LST, SSF, and LoRA across diverse datasets.]
• research_questions:[
• Can PEFT, specifically Conv-LoRA, restore and enhance SAM’s ability to learn high-level semantic information while preserving its segmentation knowledge?
• Does injecting multi-scale local priors via MoE-guided Conv-LoRA improve binary and multi-class semantic segmentation across natural images, agriculture, remote sensing, and medical datasets?
• How does Conv-LoRA compare to other PEFT methods in terms of performance, parameter overhead, and training efficiency?
• Is an end-to-end SAM feasible for segmentation tasks when prompting is kept constant and a multi-class decoder branch is added?]
• key_findings:[
• Conv-LoRA consistently outperforms other PEFT methods across natural images, agriculture, remote sensing, and healthcare benchmarks.
• Conv-LoRA adds negligible parameter overhead compared to LoRA while delivering clear performance gains.
• MoE-based dynamic scale selection yields training speedups and memory savings versus multi-scale fusion.
• Fine-tuning the image encoder (even with PEFT) is more beneficial for segmentation quality (mIoU, Dice) than decoder-only tuning.
• SAM’s pretraining on binary mask prediction limits high-level semantic learning, which Conv-LoRA helps recover.
• End-to-end adaptation of SAM for multi-class segmentation is achieved with a simple architectural modification and PEFT.]
• table_headers: [],
• table_rows: []} } } 0 4 6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

실험 결과

연구 질문

• RQ1 Can PEFT, specifically Conv-LoRA, restore and enhance SAM’s ability to learn high-level semantic information while preserving its segmentation knowledge?
• RQ2 Does injecting multi-scale local priors via MoE-guided Conv-LoRA improve binary and multi-class semantic segmentation across natural images, agriculture, remote sensing, and medical datasets?
• RQ3 How does Conv-LoRA compare to other PEFT methods in terms of performance, parameter overhead, and training efficiency?
• RQ4 Is an end-to-end SAM feasible for segmentation tasks when prompting is kept constant and a multi-class decoder branch is added?

주요 결과

• Conv-LoRA consistently outperforms other PEFT methods across natural images, agriculture, remote sensing, and healthcare benchmarks.
• Conv-LoRA adds negligible parameter overhead compared to LoRA while delivering clear performance gains.
• MoE-based dynamic scale selection yields training speedups and memory savings versus multi-scale fusion.
• Fine-tuning the image encoder (even with PEFT) is more beneficial for segmentation quality (mIoU, Dice) than decoder-only tuning.
• SAM의 프리트레이닝이 이진 마스크 예측에 편향되어 있어 고수준 시맨틱 학습이 제한되지만 Conv-LoRA가 이를 회복하는 데 도움을 준다.
• 엔드-투-엔드 SAM의 다중 클래스 세그멘테이션 적응은 간단한 구조 수정과 PEFT로 달성된다.