[Paper Review] CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarios

Jingyang Lin, Yingda Xia+6 more|arXiv (Cornell University)|Apr 23, 2024

Radiomics and Machine Learning in Medical ImagingMedicine3 citations

TL;DR

CT-GLIP introduces a novel 3D grounded vision-language pretraining framework for full-body CT scans and radiology reports, using organ-level image-text pairs and an abnormality dictionary to enhance contrastive learning. It achieves state-of-the-art zero-shot and fine-tuning performance in organ and abnormality recognition, outperforming vanilla CLIP by up to 7.7% in AUC for cancer detection on non-contrast CT scans.

ABSTRACT

3D medical vision-language (VL) pretraining has shown potential in radiology by leveraging large-scale multimodal datasets with CT-report pairs. However, existing methods primarily rely on a global VL alignment directly adapted from 2D scenarios. The entire 3D image is transformed into one global embedding, resulting in a loss of sparse but critical semantics essential for accurately aligning with the corresponding diagnosis. To address this limitation, we propose CT-GLIP, a 3D Grounded Language-Image Pretrained model that constructs fine-grained CT-report pairs to enhance extit{grounded} cross-modal contrastive learning, effectively aligning grounded visual features with precise textual descriptions. Leveraging the grounded cross-modal alignment, CT-GLIP improves performance across diverse downstream tasks and can even identify organs and abnormalities in a zero-shot manner using natural language. CT-GLIP is trained on a multimodal CT dataset comprising 44,011 organ-level CT-report pairs from 17,702 patients, covering 104 organs. Evaluation is conducted on four downstream tasks: zero-shot organ recognition (OR), zero-shot abnormality detection (AD), tumor detection (TD), and tumor segmentation (TS). Empirical results show that it outperforms its counterparts with global VL alignment. Compared to vanilla CLIP, CT-GLIP achieves average performance improvements of 15.1% of F1 score, 1.9% of AUC, and 3.2% of DSC for zero-shot AD, TD, and TS tasks, respectively. This study highlights the significance of grounded VL alignment in enabling 3D medical VL foundation models to understand sparse representations within CT scans.

Motivation & Objective

• To extend medical vision-language pretraining (Med-VLP) from 2D images to 3D full-body CT scans, which are more complex and clinically relevant but underexplored due to data sparsity and representation challenges.
• To address the difficulty of aligning sparse 3D visual representations with precise textual descriptions by introducing organ-level grounded image-text pairs.
• To improve contrastive learning in 3D Med-VLP by increasing the diversity and availability of negative contrastive pairs through an abnormality dictionary.
• To validate the effectiveness of grounded image-text alignment in zero-shot and fine-tuning scenarios for organ classification, abnormality detection, and tumor segmentation across multiple organs and cancer types.
• To establish a benchmark dataset and evaluation protocol for 3D Med-VLP using 44,011 organ-level image-report pairs from 17,702 patients across 104 organs.

Proposed method

• The method constructs organ-level image-text pairs by segmenting 104 organs in 3D CT scans using TotalSegmentator, enabling precise visual-textual grounding.
• Radiology reports are parsed using LLaMA-2 and manual validation to extract organ-specific diagnostic descriptions, creating concise, grounded textual components.
• Two contrastive learning objectives are introduced: organ-text alignment for basic anatomical concept understanding and abnormality-text alignment for zero-shot detection of pathologies.
• An abnormality dictionary is constructed to generate diverse negative text descriptions, increasing the number and variety of contrastive pairs and improving representation learning.
• The framework is trained using multimodal contrastive loss with both CNN and Vision Transformer (ViT) backbones, enabling transfer to downstream tasks like tumor segmentation and cancer screening.
• The model is evaluated on zero-shot organ and abnormality classification and fine-tuned on a dedicated cancer screening dataset with 700 non-contrast CT scans and histopathology-confirmed tumor masks.

Experimental results

Research questions

• RQ1 Can grounded 3D vision-language pretraining with organ-level image-text pairs improve zero-shot recognition of organs and abnormalities in full-body CT scans?
• RQ2 How does the use of an abnormality dictionary enhance contrastive learning in 3D Med-VLP, especially given the limited number of training samples?
• RQ3 Does the proposed CT-GLIP framework outperform standard whole-image report alignment and vanilla CLIP in both zero-shot and fine-tuning settings for 3D medical imaging?
• RQ4 To what extent can CT-GLIP improve downstream performance in multi-cancer screening tasks such as tumor segmentation and detection on non-contrast CT scans?
• RQ5 Can the model generalize to rare or unseen abnormalities in a zero-shot manner using natural language descriptions?

Key findings

• CT-GLIP achieves a 71.90% zero-shot abnormality detection F1-score on a test set of 1,130 patients, outperforming vanilla CLIP by 2.63% in the ViT-based setting.
• In fine-tuning for tumor segmentation, CT-GLIP improves the mean Dice score by 4.8% over training from scratch and 1.3% over fine-tuning with vanilla CLIP on the nnUNet backbone.
• For cancer screening, CT-GLIP achieves a 7.4% higher mean AUC score than training from scratch and a 2.2% improvement over fine-tuning with vanilla CLIP on the nnUNet backbone.
• On the MiT-based backbone, CT-GLIP improves mean tumor segmentation Dice score by 13.1% over training from scratch and 5.1% over fine-tuning with vanilla CLIP.
• For cancer detection, CT-GLIP achieves a 7.7% higher mean AUC than training from scratch and a 1.6% improvement over fine-tuning with vanilla CLIP on the MiT backbone.
• The model demonstrates strong zero-shot generalization, correctly identifying organs and abnormalities in 86.24% of cases using only natural language prompts without fine-tuning.