[Paper Review] EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Model

Yuxuan Zhang, Tianheng Cheng+8 more|arXiv (Cornell University)|Jun 28, 2024

Semantic Web and Ontologies4 citations

TL;DR

EVF-SAM proposes a simple yet effective method to enable the Segment Anything Model (SAM) for text-prompted referring expression segmentation by fusing vision and language features early in a multimodal encoder, such as BEIT-3. The approach achieves state-of-the-art performance on RefCOCO/+/g benchmarks with 82% fewer parameters than prior LLM-based methods, demonstrating that early vision-language fusion significantly improves segmentation accuracy over text-only or late-fusion approaches.

ABSTRACT

Segment Anything Model (SAM) has attracted widespread attention for its superior interactive segmentation capabilities with visual prompts while lacking further exploration of text prompts. In this paper, we empirically investigate what text prompt encoders (e.g., CLIP or LLM) are good for adapting SAM for referring expression segmentation and introduce the Early Vision-language Fusion-based SAM (EVF-SAM). EVF-SAM is a simple yet effective referring segmentation method which exploits multimodal prompts (i.e., image and text) and comprises a pre-trained vision-language model to generate referring prompts and a SAM model for segmentation. Surprisingly, we observe that: (1) multimodal prompts and (2) vision-language models with early fusion (e.g., BEIT-3) are beneficial for prompting SAM for accurate referring segmentation. Our experiments show that the proposed EVF-SAM based on BEIT-3 can obtain state-of-the-art performance on RefCOCO/+/g for referring expression segmentation and demonstrate the superiority of prompting SAM with early vision-language fusion. In addition, the proposed EVF-SAM with 1.32B parameters achieves remarkably higher performance while reducing nearly 82% of parameters compared to previous SAM methods based on large multimodal models.

Motivation & Objective

• To address the lack of effective text-prompted segmentation in the Segment Anything Model (SAM), which currently supports only visual prompts like points and boxes.
• To investigate which text prompt encoders—such as CLIP or large language models (LLMs)—are most effective for adapting SAM to referring expression segmentation (RES).
• To explore whether multimodal prompts (image + text) and early vision-language fusion improve SAM's performance in RES tasks.
• To develop a lightweight, efficient, and end-to-end framework that avoids complex instruction templates and high computational costs of LLM fine-tuning.
• To demonstrate that early fusion in vision-language models (e.g., BEIT-3) outperforms late fusion or standalone text encoders for prompting SAM with natural language descriptions.

Proposed method

• EVF-SAM integrates a pre-trained multimodal encoder with early vision-language fusion (e.g., BEIT-3) to generate joint image-text embeddings as prompts for SAM.
• The method uses a simple projector head to transform the multimodal embeddings into compatible prompt embeddings for SAM’s prompt encoder.
• The model is trained end-to-end on referring expression segmentation datasets such as RefCOCO, RefCOCO+, and RefCOCOg, using standard cross-entropy loss.
• The framework leverages off-the-shelf foundation models (SAM and BEIT-3) without architectural modifications, enabling scalability and parameter efficiency.
• The approach supports both short and long-form referring expressions, including spatial and descriptive language, by leveraging the rich multimodal representations of early-fused encoders.
• A lightweight variant, EVF-SAM with Efficient-SAM, reduces model size by 600M parameters while maintaining strong performance.

Experimental results

Research questions

• RQ1 Does using multimodal prompts (image and text) improve SAM's performance in referring expression segmentation compared to text-only prompts?
• RQ2 Is early vision-language fusion in the prompt encoder more effective than late fusion or standalone text encoders for prompting SAM with natural language?
• RQ3 Can a lightweight vision-language model (e.g., BEIT-3) outperform large language models (LLMs) in prompting SAM for referring segmentation, despite having far fewer parameters?
• RQ4 How does the choice of multimodal encoder (e.g., CLIP, ViLT, BEIT-3) affect the segmentation accuracy of text-prompted SAM?
• RQ5 To what extent can EVF-SAM generalize across different SAM variants (e.g., SAM-Huge vs. Efficient-SAM) and benchmarks (RefCOCO/+/g)?

Key findings

• EVF-SAM with BEIT-3-Large achieves state-of-the-art performance on RefCOCO/+/g benchmarks, with an average mIoU of 78.0, surpassing previous LLM-based methods.
• The model reduces parameters by 82% compared to LISA (a prior SOTA LLM-based method), achieving 1.32B parameters while outperforming larger models.
• BEIT-3-Base, a smaller encoder, leads to a significant performance drop (e.g., 71.6 avg mIoU), confirming that model capacity of the multimodal encoder is critical.
• EVF-SAM outperforms CLIP-based prompting by 12.7% average mIoU on RefCOCOg, demonstrating superior understanding of long and complex referring expressions.
• The method achieves better performance with less training data and without relying on handcrafted instruction templates, unlike LLM-based approaches.
• Efficient-SAM-S variant in EVF-SAM shows negligible performance drop compared to SAM-Huge, proving the framework’s compatibility and efficiency across SAM variants.