[Paper Review] When More is Less: Incorporating Additional Datasets Can Hurt Performance By Introducing Spurious Correlations

Rhys Compton, Lily Zhang+2 more|arXiv (Cornell University)|Aug 8, 2023

Radiomics and Machine Learning in Medical Imaging4 citations

TL;DR

This paper demonstrates that adding external medical imaging datasets can degrade model performance due to spurious correlations between disease labels and hospital-specific imaging artifacts. In 43% of evaluated settings, models trained on combined data from two hospitals performed worse on worst-group accuracy than models trained on a single hospital’s data, despite improved distributional similarity to test data.

ABSTRACT

In machine learning, incorporating more data is often seen as a reliable strategy for improving model performance; this work challenges that notion by demonstrating that the addition of external datasets in many cases can hurt the resulting model's performance. In a large-scale empirical study across combinations of four different open-source chest x-ray datasets and 9 different labels, we demonstrate that in 43% of settings, a model trained on data from two hospitals has poorer worst group accuracy over both hospitals than a model trained on just a single hospital's data. This surprising result occurs even though the added hospital makes the training distribution more similar to the test distribution. We explain that this phenomenon arises from the spurious correlation that emerges between the disease and hospital, due to hospital-specific image artifacts. We highlight the trade-off one encounters when training on multiple datasets, between the obvious benefit of additional data and insidious cost of the introduced spurious correlation. In some cases, balancing the dataset can remove the spurious correlation and improve performance, but it is not always an effective strategy. We contextualize our results within the literature on spurious correlations to help explain these outcomes. Our experiments underscore the importance of exercising caution when selecting training data for machine learning models, especially in settings where there is a risk of spurious correlations such as with medical imaging. The risks outlined highlight the need for careful data selection and model evaluation in future research and practice.

Motivation & Objective

• To investigate whether incorporating additional datasets always improves model generalization in medical imaging.
• To identify conditions under which adding data leads to performance degradation due to spurious correlations.
• To evaluate the effectiveness of data balancing techniques in mitigating such degradation.
• To provide practical guidance for data selection and model evaluation in multi-source medical AI.

Proposed method

• Empirically evaluate models trained on single-hospital vs. multi-hospital chest X-ray datasets across 9 disease labels and 4 hospitals.
• Use worst-group accuracy as the primary evaluation metric to detect performance disparities across hospitals.
• Train models with and without data balancing via undersampling to reduce spurious correlations between hospital and disease.
• Analyze learned representations to assess whether models encode hospital-specific signals even without explicit supervision.
• Apply theoretical tools from Puli et al. (2022) to explain why balancing does not always yield robust models.
• Compare results across different model architectures and training configurations to isolate the effect of data source mixing.

Figure 1: High level overview of experiments. We study changes in worst-group accuracy (bold) when training on additional labelled data from an external hospital.

Experimental results

Research questions

• RQ1 Does adding data from an external hospital always improve worst-group accuracy in chest X-ray classification?
• RQ2 What role do hospital-specific imaging artifacts play in introducing spurious correlations between disease labels and data sources?
• RQ3 Can data balancing via undersampling effectively eliminate performance degradation caused by multi-source training?
• RQ4 Under what conditions does multi-source training lead to worse generalization than single-source training?
• RQ5 How do model representations encode hospital identity, even when not explicitly trained to do so?

Key findings

• In 43% of the evaluated settings, models trained on data from two hospitals had lower worst-group accuracy than models trained on a single hospital’s data.
• The performance drop occurred even when the additional data made the training distribution more similar to the test distribution, indicating that distributional similarity alone is insufficient for generalization.
• Models trained on multi-source data learned to use hospital identity as a shortcut for disease prediction, with representations capable of perfectly distinguishing hospitals not seen during training.
• Data balancing via undersampling improved performance in some cases but failed to improve worst-group accuracy in 43% of settings where multi-source training hurt performance.
• Balancing did not guarantee better performance than training on a single source, and in some cases, it led to worse outcomes than not including the additional dataset at all.
• Theoretical analysis confirms that balancing is not a guaranteed solution to spurious correlation issues, especially when the underlying data distribution is complex.

Figure 2: Change in worst-group performance after including a second training dataset. We see both improvements in performance as expected, but also many deteriorations. For every element in the legend, the first dataset is the base environment, and the second is the added environment; for instance,