[Paper Review] FluoroSAM: A Language-promptable Foundation Model for Flexible X-ray Image Segmentation

Benjamin D. Killeen, Liam J. Wang+6 more|arXiv (Cornell University)|Mar 12, 2024

Radiomics and Machine Learning in Medical Imaging6 citations

TL;DR

FluoroSAM is a language-aligned SAM-style foundation model trained from synthetic X-ray data to enable text-prompted, zero-shot, and point-refined segmentation of X-ray images, outperforming existing SAM variants on real fluoroscopic data and even generalizing to unseen tasks like full-lung segmentation in chest X-rays.

ABSTRACT

Language promptable X-ray image segmentation would enable greater flexibility for human-in-the-loop workflows in diagnostic and interventional precision medicine. Prior efforts have contributed task-specific models capable of solving problems within a narrow scope, but expanding to broader use requires additional data, annotations, and training time. Recently, language-aligned foundation models (LFMs) -- machine learning models trained on large amounts of highly variable image and text data thus enabling broad applicability -- have emerged as promising tools for automated image analysis. Existing foundation models for medical image analysis focus on scenarios and modalities where large, richly annotated datasets are available. However, the X-ray imaging modality features highly variable image appearance and applications, from diagnostic chest X-rays to interventional fluoroscopy, with varying availability of data. To pave the way toward an LFM for comprehensive and language-aligned analysis of arbitrary medical X-ray images, we introduce FluoroSAM, a language-promptable variant of the Segment Anything Model, trained from scratch on 3M synthetic X-ray images from a wide variety of human anatomies, imaging geometries, and viewing angles. These include pseudo-ground truth masks for 128 organ types and 464 tools with associated text descriptions. FluoroSAM is capable of segmenting myriad anatomical structures and tools based on natural language prompts, thanks to the novel incorporation of vector quantization (VQ) of text embeddings in the training process. We demonstrate FluoroSAM's performance quantitatively on real X-ray images and showcase on several applications how FluoroSAM is a key enabler for rich human-machine interaction in the X-ray image acquisition and analysis context. Code is available at https://github.com/arcadelab/fluorosam.

Motivation & Objective

• Provide automated and interactive segmentation for arbitrary X-ray images without task-specific retraining.
• Leverage a language-aligned prompt paradigm to handle overlapping X-ray structures and ambiguities.
• Create a large-scale synthetic X-ray dataset with organ and device masks to train a foundation model from scratch.
• Demonstrate zero-shot generalization to unseen X-ray classes and real-world applicability in cadaveric data and chest X-rays.

Proposed method

• Train a SAM-style foundation model from scratch on a synthetic X-ray dataset (1.6M DRRs) with 63M masks across 128 organs and 464 devices.
• Use text prompts augmented by a large language model to describe targets and guide segmentation.
• Incorporate point-based refinement prompts to improve initial text-prompted masks.
• Employ MedCLIP text embeddings (frozen) and an EfficientViT image encoder, with a GPT-3.5-based augmentation loop for prompt variations.
• Apply domain randomization and sim-to-real transfer techniques to bridge synthetic-real gaps.

Experimental results

Research questions

• RQ1 Can a language-aligned foundation model segment arbitrary objects and anatomy in X-ray images using text prompts alone?
• RQ2 Does incorporating point-based refinements improve segmentation quality on real X-ray data?
• RQ3 To what extent can the model generalize to unseen X-ray classes (zero-shot) such as whole-lung segmentation in chest X-rays?

Key findings

• FluoroSAM achieves 0.43±0.26 Dice on the synthetic test set with text prompts alone, improving to 0.85±0.11 with point-based refinement.
• With two points, FluoroSAM reaches 0.68±0.20 Dice, outperforming MedSAM (0.60±0.22) and SAM (0.58±0.19).
• In a fluoroscopic cadaver study, text prompts yield 0.39 Dice for hard tissues and 0.26 overall, while adding point prompts yields 0.90±0.15 for hard and 0.73±0.15 for soft tissues (better than SAM/MedSAM baselines).
• Zero-shot whole-lung segmentation on chest X-rays (no training on full-lung class) achieves 0.52±0.21 Dice with text prompts, rising to 0.90±0.04 with point refinement.
• FluoroSAM demonstrates zero-shot generalization and interactive refinement capabilities in real X-ray imagery, including cadaveric and chest X-ray scenarios.