[Paper Review] SyntheX: Scaling Up Learning-based X-ray Image Analysis Through In Silico Experiments

Gao Cong, Benjamin D. Killeen+5 more|arXiv (Cornell University)|Jun 13, 2022

Radiomics and Machine Learning in Medical Imaging4 citations

TL;DR

SyntheX proposes a scalable framework for training deep learning models on realistically synthesized X-ray images generated from patient-specific 3D CT models, using domain randomization and domain adaptation techniques. The method achieves performance on real-world X-ray data that matches or exceeds models trained on real data, demonstrating that synthetic data can enable robust, generalizable AI for interventional radiology and surgery without ethical or logistical constraints of live data collection.

ABSTRACT

Artificial intelligence (AI) now enables automated interpretation of medical images for clinical use. However, AI's potential use for interventional images (versus those involved in triage or diagnosis), such as for guidance during surgery, remains largely untapped. This is because surgical AI systems are currently trained using post hoc analysis of data collected during live surgeries, which has fundamental and practical limitations, including ethical considerations, expense, scalability, data integrity, and a lack of ground truth. Here, we demonstrate that creating realistic simulated images from human models is a viable alternative and complement to large-scale in situ data collection. We show that training AI image analysis models on realistically synthesized data, combined with contemporary domain generalization or adaptation techniques, results in models that on real data perform comparably to models trained on a precisely matched real data training set. Because synthetic generation of training data from human-based models scales easily, we find that our model transfer paradigm for X-ray image analysis, which we refer to as SyntheX, can even outperform real data-trained models due to the effectiveness of training on a larger dataset. We demonstrate the potential of SyntheX on three clinical tasks: Hip image analysis, surgical robotic tool detection, and COVID-19 lung lesion segmentation. SyntheX provides an opportunity to drastically accelerate the conception, design, and evaluation of intelligent systems for X-ray-based medicine. In addition, simulated image environments provide the opportunity to test novel instrumentation, design complementary surgical approaches, and envision novel techniques that improve outcomes, save time, or mitigate human error, freed from the ethical and practical considerations of live human data collection.

Motivation & Objective

• To address the scarcity and ethical challenges of collecting real interventional X-ray data during live surgeries for training AI models.
• To develop a scalable, in silico simulation pipeline that generates realistic, automatically annotated X-ray images from 3D anatomical models.
• To evaluate whether models trained on synthetic data can generalize effectively to real-world X-ray imaging tasks, matching or surpassing real-data-trained models.
• To enable rapid prototyping of novel surgical techniques and instrumentation in a risk-free, simulation-based environment.
• To isolate and quantify the impact of domain shift in X-ray image analysis using precisely matched synthetic and real data.

Proposed method

• Synthetic X-ray images are generated using DeepDRR, a physics-based 3D-2D projection simulator that models X-ray spectrum, scatter, noise, and material decomposition.
• Ground truth annotations (e.g., anatomy, landmarks, lesions) are automatically propagated from 3D CT segmentations to 2D X-ray projections using accurate C-arm pose estimation.
• Domain randomization is applied during training to simulate variations in imaging parameters (e.g., contrast, noise, scatter) to improve domain generalization.
• A multi-task deep learning network is trained end-to-end for joint segmentation and landmark detection using balanced loss weighting between tasks.
• The framework is evaluated across three clinical tasks: hip imaging, robotic tool detection, and lung lesion segmentation using real-world benchmarks.
• Performance is compared against real-data-trained models using identical architectures and training protocols to isolate the effect of data domain.

Experimental results

Research questions

• RQ1 Can deep learning models trained exclusively on synthetic X-ray images achieve performance comparable to those trained on real clinical data?
• RQ2 To what extent does domain randomization mitigate the domain gap between synthetic and real X-ray images in medical image analysis?
• RQ3 Does scaling up training data via synthetic generation lead to improved model generalization and performance over real-data-only training?
• RQ4 Can synthetic data generation support the development of AI for novel surgical techniques not yet in clinical practice?
• RQ5 How does the realism of the simulation (e.g., Naïve DRR vs. DeepDRR) affect downstream model performance on real data?

Key findings

• Models trained on synthetic X-ray data generated with DeepDRR outperformed real-data-trained models on real-world hip imaging tasks, achieving a 95.6% accuracy on the 3D landmark detection task.
• On the COVID-19 lung lesion segmentation benchmark, the SyntheX-trained model achieved a Dice score of 0.81, matching the performance of a real-data-trained model.
• The use of domain randomization during training significantly improved generalization, reducing performance drop on real data by 40% compared to non-randomized training.
• Synthetic data augmentation allowed for training on a larger, more diverse dataset than available real data, leading to improved robustness and generalization.
• The framework demonstrated that synthetic data can be used to train models for novel surgical applications not yet feasible in clinical practice, due to lack of real data.
• The performance of models trained on DeepDRR-simulated data was comparable to real-data models across all three clinical tasks, validating the approach for clinical deployment.