[Paper Review] The Impact of Scanner Domain Shift on Deep Learning Performance in Medical Imaging: an Experimental Study

Brian H.W. Guo, Darui Lu+5 more|arXiv (Cornell University)|Sep 6, 2024

Radiomics and Machine Learning in Medical ImagingMedicine3 citations

TL;DR

This study systematically evaluates scanner domain shift in deep learning for medical imaging across MRI, CT, and X-ray modalities. It demonstrates that performance drops significantly—most severely in MRI, moderately in X-ray, and slightly in CT—due to differences in scanner parameters and post-processing, highlighting the need to account for scanner variability in model development and deployment for clinical use.

ABSTRACT

Purpose: Medical images acquired using different scanners and protocols can differ substantially in their appearance. This phenomenon, scanner domain shift, can result in a drop in the performance of deep neural networks which are trained on data acquired by one scanner and tested on another. This significant practical issue is well-acknowledged, however, no systematic study of the issue is available across different modalities and diagnostic tasks. Materials and Methods: In this paper, we present a broad experimental study evaluating the impact of scanner domain shift on convolutional neural network performance for different automated diagnostic tasks. We evaluate this phenomenon in common radiological modalities, including X-ray, CT, and MRI. Results: We find that network performance on data from a different scanner is almost always worse than on same-scanner data, and we quantify the degree of performance drop across different datasets. Notably, we find that this drop is most severe for MRI, moderate for X-ray, and quite small for CT, on average, which we attribute to the standardized nature of CT acquisition systems which is not present in MRI or X-ray. We also study how injecting varying amounts of target domain data into the training set, as well as adding noise to the training data, helps with generalization. Conclusion: Our results provide extensive experimental evidence and quantification of the extent of performance drop caused by scanner domain shift in deep learning across different modalities, with the goal of guiding the future development of robust deep learning models for medical image analysis.

Motivation & Objective

• To investigate the impact of scanner domain shift on deep learning performance across multiple medical imaging modalities.
• To quantify the extent of performance degradation when models trained on one scanner are tested on another.
• To assess the effectiveness of data injection and data augmentation in mitigating domain shift effects.
• To understand modality-specific differences in susceptibility to scanner domain shift, particularly related to acquisition standardization.
• To provide empirical guidance for developing robust deep learning models in real-world clinical settings with heterogeneous scanners.

Proposed method

• Trained convolutional neural networks on source-domain data from one scanner and evaluated performance on target-domain data from different scanners.
• Used seven datasets spanning MRI, CT, and X-ray modalities across different anatomical regions and diagnostic tasks.
• Injected increasing proportions of target-domain data into the training set to evaluate domain generalization.
• Added Gaussian noise to training images to assess its effect on out-of-domain performance.
• Compared performance across different scanner manufacturers and models, focusing on image acquisition parameters and post-processing differences.
• Analyzed image feature distributions and their sensitivity to scanner-specific parameters such as TR, TE, and slice thickness.

Figure 1: Seeded Training and Testing Procedure: See Section 3.1 for more details.

Experimental results

Research questions

• RQ1 How does scanner domain shift affect deep learning performance across different medical imaging modalities (MRI, CT, X-ray)?
• RQ2 What is the magnitude of performance degradation when models trained on one scanner are tested on data from a different scanner?
• RQ3 How does the inclusion of target-domain data during training affect model generalization across scanners?
• RQ4 Does adding noise to training data improve generalization to out-of-domain scanner data?
• RQ5 Why is MRI more susceptible to scanner domain shift than CT, despite both being MRI and CT modalities?

Key findings

• In 13 out of 14 evaluated settings (dataset, task, source/target domains), model performance dropped when tested on data from a different scanner.
• MRI exhibited the most severe performance degradation on average, followed by X-ray, with CT showing the smallest drop, attributed to higher standardization in CT acquisition protocols.
• The performance drop was most pronounced in MRI due to high variability in parameters such as echo time (TE), repetition time (TR), and flip angle across scanner manufacturers.
• Injecting target-domain data into training improved generalization, but the extent varied by modality and task, with MRI showing the greatest benefit from such data.
• Adding Gaussian noise to training data did not improve out-of-domain performance, indicating that simple data augmentation is insufficient for domain shift mitigation.
• Scanner-specific post-processing and acquisition parameters significantly alter image features, leading to model performance loss even when image differences are visually subtle.

Figure 2: Visualization of the change in network performance due to the test scanner domain shifting away from the training scanner domain (see Table 2 for reference). Each pair of bars is for a single modality and training set scanner manufacturer.