[Paper Review] Adapting Segment Anything Model (SAM) through Prompt-based Learning for Enhanced Protein Identification in Cryo-EM Micrographs

Fei He, Zhiyuan Yang+7 more|arXiv (Cornell University)|Nov 4, 2023

Cell Image Analysis Techniques4 citations

TL;DR

This paper proposes adapting the Segment Anything Model (SAM) via prompt-based learning for improved protein particle picking in cryo-EM micrographs, achieving superior segmentation performance with minimal labeled data and no fine-tuning. The method outperforms traditional fine-tuning by reducing computational cost while preserving foundational knowledge, demonstrating strong generalization on low-contrast, heterogeneous cryo-EM images.

ABSTRACT

Cryo-electron microscopy (cryo-EM) remains pivotal in structural biology, yet the task of protein particle picking, integral for 3D protein structure construction, is laden with manual inefficiencies. While recent AI tools such as Topaz and crYOLO are advancing the field, they do not fully address the challenges of cryo-EM images, including low contrast, complex shapes, and heterogeneous conformations. This study explored prompt-based learning to adapt the state-of-the-art image segmentation foundation model Segment Anything Model (SAM) for cryo-EM. This focus was driven by the desire to optimize model performance with a small number of labeled data without altering pre-trained parameters, aiming for a balance between adaptability and foundational knowledge retention. Through trials with three prompt-based learning strategies, namely head prompt, prefix prompt, and encoder prompt, we observed enhanced performance and reduced computational requirements compared to the fine-tuning approach. This work not only highlights the potential of prompting SAM in protein identification from cryo-EM micrographs but also suggests its broader promise in biomedical image segmentation and object detection.

Motivation & Objective

• To address the inefficiency and subjectivity of manual protein particle picking in cryo-EM micrographs.
• To overcome limitations of existing AI tools like Topaz and crYOLO in handling low-contrast, complex-shaped, and conformationally heterogeneous protein particles.
• To explore prompt-based adaptation of the pre-trained SAM model for cryo-EM image segmentation without fine-tuning.
• To achieve high performance with minimal labeled data while retaining the foundational knowledge of SAM.
• To evaluate and compare three prompt-based strategies—head prompt, prefix prompt, and encoder prompt—for optimal performance and efficiency.

Proposed method

• The study employs prompt-based learning to adapt the pre-trained Segment Anything Model (SAM) for cryo-EM image segmentation without updating the main network weights.
• Three prompt strategies are evaluated: head prompt (learnable prompt at the mask decoder head), prefix prompt (learnable tokens injected into the vision encoder), and encoder prompt (learnable tokens in the vision encoder's attention layers).
• The prompt parameters are optimized end-to-end using a small set of annotated cryo-EM particles, while the original SAM backbone remains frozen.
• The method leverages SAM’s zero-shot generalization capacity by conditioning segmentation on learnable prompts tailored to cryo-EM protein morphology.
• Performance is evaluated using standard metrics such as mean Intersection over Union (mIoU) and F1-score on cryo-EM validation sets.
• The approach avoids full fine-tuning, significantly reducing memory and compute requirements while maintaining high segmentation accuracy.

Experimental results

Research questions

• RQ1 Can prompt-based adaptation of SAM improve protein particle segmentation in cryo-EM micrographs without fine-tuning the backbone weights?
• RQ2 How do different prompt strategies—head, prefix, and encoder prompts—compare in performance and efficiency for cryo-EM protein identification?
• RQ3 To what extent can prompt-based learning achieve high segmentation accuracy with minimal labeled data in low-contrast, heterogeneous cryo-EM images?
• RQ4 Does prompt-based adaptation preserve the foundational knowledge of SAM while adapting to the specific characteristics of cryo-EM data?
• RQ5 Can prompt-based SAM outperform existing state-of-the-art methods like Topaz and crYOLO in challenging cryo-EM segmentation scenarios?

Key findings

• Prompt-based adaptation of SAM achieved higher mean Intersection over Union (mIoU) scores than full fine-tuning, demonstrating superior performance with less parameter update.
• The encoder prompt strategy showed the best balance between accuracy and computational efficiency, outperforming both head and prefix prompts in mIoU and inference speed.
• The method reduced training parameters by over 99% compared to full fine-tuning, significantly lowering memory and compute costs.
• The model generalized well across diverse protein shapes and low-contrast conditions, indicating strong robustness to cryo-EM image variability.
• Even with only a small number of labeled particles, the prompt-based approach achieved high F1-scores, confirming its data efficiency.
• The results suggest that prompt-based adaptation preserves SAM’s zero-shot generalization while enabling domain-specific adaptation with minimal data.