[Paper Review] CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection

Jie Liu, Yixiao Zhang+8 more|arXiv (Cornell University)|Jan 2, 2023

Radiomics and Machine Learning in Medical ImagingMedicine18 citations

TL;DR

Introduces a CLIP-powered universal model that encodes anatomical relationships via text embeddings to segment 25 organs and detect 6 tumor types across 14 public datasets, achieving state-of-the-art results and strong generalization.

ABSTRACT

An increasing number of public datasets have shown a marked impact on automated organ segmentation and tumor detection. However, due to the small size and partially labeled problem of each dataset, as well as a limited investigation of diverse types of tumors, the resulting models are often limited to segmenting specific organs/tumors and ignore the semantics of anatomical structures, nor can they be extended to novel domains. To address these issues, we propose the CLIP-Driven Universal Model, which incorporates text embedding learned from Contrastive Language-Image Pre-training (CLIP) to segmentation models. This CLIP-based label encoding captures anatomical relationships, enabling the model to learn a structured feature embedding and segment 25 organs and 6 types of tumors. The proposed model is developed from an assembly of 14 datasets, using a total of 3,410 CT scans for training and then evaluated on 6,162 external CT scans from 3 additional datasets. We rank first on the Medical Segmentation Decathlon (MSD) public leaderboard and achieve state-of-the-art results on Beyond The Cranial Vault (BTCV). Additionally, the Universal Model is computationally more efficient (6x faster) compared with dataset-specific models, generalized better to CT scans from varying sites, and shows stronger transfer learning performance on novel tasks.

Motivation & Objective

• Address partial labeling and label inconsistency across multiple public abdominal datasets.
• Leverage CLIP text embeddings to encode anatomical relationships for improved segmentation and tumor detection.
• Develop a universal model architecture that supports multiple backbones and generalizes to external CT scans without fine-tuning.
• Demonstrate state-of-the-art performance on MSD and BTCV benchmarks and provide transferability to downstream tasks.

Proposed method

• Assemble 14 public datasets totaling 3,410 CT scans with 25 partially annotated organs and 6 tumors.
• Use a CLIP-based text embedding as an anatomical prompt to guide segmentation, replacing one-hot labels.
• Construct a text branch that generates class-specific parameters via an MLP conditioning on CLIP embeddings and global image features.
• Process CT scans with standardized preprocessing and apply a vision encoder followed by a 1x1x1 conv-based text-driven segmentor.
• Implement masked back-propagation to train only on classes with available labels in partially labeled data.
• Evaluate with Dice Similarity Coefficient (DSC), Normalized Surface Distance (NSD), sensitivity, specificity, and harmonic mean for tumor detection.

Experimental results

Research questions

• RQ1 Can CLIP-based label embeddings capture anatomical relationships to improve multi-organ segmentation and tumor detection across partially labeled datasets?
• RQ2 Does a universal model leveraging text prompts generalize better to external CT data and diverse scanners compared to dataset-specific models?
• RQ3 How does masked back-propagation affect learning with partial labels and data inconsistency?
• RQ4 What is the impact of different medical prompts on CLIP embeddings for segmentation performance?
• RQ5 Can the universal model serve as an effective foundation model for downstream medical imaging tasks?

Key findings

• The CLIP-Driven Universal Model achieves state-of-the-art performance on MSD and BTCV benchmarks.
• The model demonstrates high organ segmentation accuracy with six organs matching intra-observer human variability.
• Tumor detection achieves high harmonic means, indicating strong sensitivity with low false positives across datasets.
• It is about 6x faster in inference than dataset-specific models, demonstrating superior efficiency.
• The framework generalizes to CT scans from various hospitals without additional tuning, and backbones can be swapped between CNNs and Transformers.
• External validation shows robust generalizability on 3D-IRCADb and JHH datasets, outperforming prior methods.