[Paper Review] Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Images

Che Liu, Anand Shah+2 more|arXiv (Cornell University)|Oct 10, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

This paper proposes training medical vision-language models exclusively on synthetic chest X-ray images generated from real radiology reports using domain-specific diffusion models, demonstrating that performance matches or exceeds real-image pre-training across classification, segmentation, and detection tasks. The key contribution is a large-scale synthetic medical image dataset paired with anonymized reports, enabling privacy-preserving, scalable pre-training without real medical images.

ABSTRACT

Medical Vision-Language Pre-training (VLP) learns representations jointly from medical images and paired radiology reports. It typically requires large-scale paired image-text datasets to achieve effective pre-training for both the image encoder and text encoder. The advent of text-guided generative models raises a compelling question: Can VLP be implemented solely with synthetic images generated from genuine radiology reports, thereby mitigating the need for extensively pairing and curating image-text datasets? In this work, we scrutinize this very question by examining the feasibility and effectiveness of employing synthetic images for medical VLP. We replace real medical images with their synthetic equivalents, generated from authentic medical reports. Utilizing three state-of-the-art VLP algorithms, we exclusively train on these synthetic samples. Our empirical evaluation across three subsequent tasks, namely image classification, semantic segmentation and object detection, reveals that the performance achieved through synthetic data is on par with or even exceeds that obtained with real images. As a pioneering contribution to this domain, we introduce a large-scale synthetic medical image dataset, paired with anonymized real radiology reports. This alleviates the need of sharing medical images, which are not easy to curate and share in practice. The code and the dataset can be found in \href{https://github.com/cheliu-computation/MedSyn-RepLearn/tree/main}{https://github.com/cheliu-computation/MedSyn-RepLearn/tree/main}.

Motivation & Objective

• To investigate whether medical vision-language pre-training (VLP) can be effectively performed using only synthetic medical images generated from real radiology reports, bypassing the need for real paired images.
• To address the critical challenge of data scarcity and privacy in medical VLP by generating high-fidelity synthetic images that preserve clinical semantics.
• To evaluate the performance of VLP models pre-trained on synthetic data across multiple downstream tasks, including image classification, semantic segmentation, and object detection.
• To introduce a large-scale, publicly available synthetic medical image dataset paired with anonymized real radiology reports to support future research.
• To compare the effectiveness of general-domain (Stable Diffusion) versus medical-domain-specific (RoentGen) generative models in supporting VLP.

Proposed method

• Text-guided image generation using two models: Stable Diffusion 2.1 (general domain) and RoentGen (medical domain-specific), with radiology report impressions as prompts.
• Synthetic images are generated at 512×512 resolution using 50 sampling steps and a fixed random seed to ensure consistency and one-to-one correspondence with reports.
• Vision-language contrastive pre-training is performed using three state-of-the-art VLP frameworks: ConVIRT, GLoRIA, and MGCA, all trained exclusively on synthetic images paired with real text.
• Pre-training is conducted using contrastive learning objectives to align image and text embeddings, with the image encoder and text encoder jointly optimized.
• Evaluation is performed on three downstream tasks: image classification (SIIM-ACR), semantic segmentation (RSNA), and object detection (RSNA and Object-CXR), using 1%, 10%, and 100% of training data.
• Performance is compared against baselines pre-trained on real images and on synthetic images from both generative models, with metrics including Dice score and mAP.

Figure 1: A) Conventional VLP pipeline with real image-text pair. B) Our VLP framework with synthetic image and real medical text. We select SD 2.1 and RoentGen [ 18 ] as the generator in this work.

Experimental results

Research questions

• RQ1 Can vision-language pre-training be effectively performed using only synthetic medical images generated from real radiology reports, without requiring real paired images?
• RQ2 How does the performance of VLP models trained on synthetic images from a general-domain diffusion model (Stable Diffusion) compare to those trained on real images or domain-specific synthetic data?
• RQ3 To what extent does using a medical-domain-specific generative model (RoentGen) improve the quality and utility of synthetic images for downstream medical vision tasks?
• RQ4 Can synthetic data mitigate the challenges of data scarcity and privacy in medical VLP while maintaining or improving model performance?
• RQ5 What is the impact of synthetic data on zero-shot and few-shot generalization in medical vision tasks?

Key findings

• VLP models pre-trained exclusively on synthetic images generated by RoentGen achieved performance on par with or better than those pre-trained on real images across all downstream tasks.
• On the SIIM-ACR classification task, ConVIRT(RoentGen) achieved a Dice score of 61.2% at 100% data, outperforming the real-image baseline (59.9%) by 1.3 percentage points.
• On the RSNA segmentation task, GLoRIA(RoentGen) achieved a Dice score of 69.8%, surpassing the real-image baseline (68.3%) by 1.5 points.
• On the RSNA object detection task, MGCA(RoentGen) achieved a mAP of 25.3% at 100% data, exceeding the real-image baseline (24.9%) by 0.4 points.
• In contrast, models pre-trained on synthetic images from Stable Diffusion (SD) showed consistent performance degradation, with ConVIRT(SD) dropping by 12.6 percentage points in Dice score on SIIM at 100% data.
• The study introduces the first large-scale synthetic CXR dataset paired with anonymized real radiology reports, publicly released upon paper acceptance, enabling privacy-preserving research.

Figure 2: The synthetic samples images generated from real radiology report from SD 2.1 and RoentGen.