[Paper Review] Synthetic Data as Validation

Qixin Hu, Alan Yuille+1 more|arXiv (Cornell University)|Oct 24, 2023

Radiomics and Machine Learning in Medical Imaging4 citations

TL;DR

This paper proposes using synthetic tumors superimposed on healthy CT volumes as a validation set to reduce overfitting and improve model generalization in liver tumor segmentation, especially for early cancer detection. By employing a continual learning framework with dynamically expanding synthetic data, the method boosts sensitivity for tiny tumors (radius <5mm) from 33.1% to 55.4% on in-domain and 33.9% to 52.3% on out-domain datasets, significantly enhancing early detection performance.

ABSTRACT

This study leverages synthetic data as a validation set to reduce overfitting and ease the selection of the best model in AI development. While synthetic data have been used for augmenting the training set, we find that synthetic data can also significantly diversify the validation set, offering marked advantages in domains like healthcare, where data are typically limited, sensitive, and from out-domain sources (i.e., hospitals). In this study, we illustrate the effectiveness of synthetic data for early cancer detection in computed tomography (CT) volumes, where synthetic tumors are generated and superimposed onto healthy organs, thereby creating an extensive dataset for rigorous validation. Using synthetic data as validation can improve AI robustness in both in-domain and out-domain test sets. Furthermore, we establish a new continual learning framework that continuously trains AI models on a stream of out-domain data with synthetic tumors. The AI model trained and validated in dynamically expanding synthetic data can consistently outperform models trained and validated exclusively on real-world data. Specifically, the DSC score for liver tumor segmentation improves from 26.7% (95% CI: 22.6%-30.9%) to 34.5% (30.8%-38.2%) when evaluated on an in-domain dataset and from 31.1% (26.0%-36.2%) to 35.4% (32.1%-38.7%) on an out-domain dataset. Importantly, the performance gain is particularly significant in identifying very tiny liver tumors (radius &lt; 5mm) in CT volumes, with Sensitivity improving from 33.1% to 55.4% on an in-domain dataset and 33.9% to 52.3% on an out-domain dataset, justifying the efficacy in early detection of cancer. The application of synthetic data, from both training and validation perspectives, underlines a promising avenue to enhance AI robustness when dealing with data from varying domains.

Motivation & Objective

• To address the challenge of limited, biased, and hard-to-annotate real-world data in medical AI, especially for early-stage tumors.
• To overcome the limitations of small, static validation sets that fail to represent out-of-domain or rare cases.
• To improve model generalization and robustness by using diverse synthetic tumors as a dynamic validation set during training.
• To enable continual learning with synthetic data to enhance detection of tiny liver tumors (radius <5mm) across in-domain and out-domain datasets.
• To demonstrate that synthetic data can outperform real data in validation, leading to better model selection and performance.

Proposed method

• Synthetic tumors are generated using a modeling-based strategy that controls shape, size, texture, location, and intensity to ensure realism.
• Healthy CT volumes are augmented with synthetic tumors to create a large, diverse validation set, replacing or supplementing real validation data.
• A continual learning framework is introduced that continuously trains on streams of out-domain data with synthetic tumors, enabling dynamic model adaptation.
• Model checkpoints are selected based on performance on synthetic validation sets, which are expanded over time to cover a wide distribution of tumor characteristics.
• The framework uses in-domain synthetic-tumor validation (e.g., from FLARE’23) to improve alignment with real test sets and enhance checkpoint selection accuracy.
• Performance is evaluated on real, fully-annotated CT volumes from multiple hospitals to assess in-domain and out-domain generalization.

Experimental results

Research questions

• RQ1 Can synthetic data serve as a reliable and diverse validation set to improve model selection in AI development, especially when real data are scarce?
• RQ2 How does using synthetic data for validation affect model generalization on in-domain and out-of-domain test sets in liver tumor segmentation?
• RQ3 To what extent can synthetic data improve the detection of tiny liver tumors (radius <5mm) in early cancer diagnosis?
• RQ4 Can a continual learning framework trained on synthetic data outperform models trained on static real data in terms of sensitivity and robustness?
• RQ5 Does in-domain synthetic-tumor validation improve the accuracy of checkpoint selection compared to out-domain or real validation sets?

Key findings

• The sensitivity for detecting tiny liver tumors (radius <5mm) improved from 33.1% to 55.4% on the in-domain test set when using synthetic data for validation.
• On the out-domain test set, sensitivity for tiny tumors increased from 33.9% to 52.3% using the synthetic validation strategy.
• The DSC score for liver tumor segmentation improved from 26.7% (95% CI: 22.6%–30.9%) to 34.5% (30.8%–38.2%) on the in-domain dataset.
• For the out-domain dataset, DSC increased from 31.1% (26.0%–36.2%) to 35.4% (32.1%–38.7%) with synthetic validation.
• The in-domain synthetic-tumor validation set accurately identified the best model checkpoint, aligning with the one selected by the real test set, proving its reliability.
• The continual learning framework with synthetic data consistently outperformed models trained and validated solely on real data, especially in detecting rare and small tumors.