[Paper Review] On the limits of cross-domain generalization in automated X-ray prediction

Joseph Cohen, Mohammad Hashir+2 more|arXiv (Cornell University)|Feb 6, 2020

Radiomics and Machine Learning in Medical Imaging35 references64 citations

TL;DR

The paper conducts a large-scale study of cross-domain generalization in chest X-ray prediction, showing that generalization issues arise from label shifts rather than image shifts, and analyzing performance, agreement, and representations across datasets.

ABSTRACT

This large scale study focuses on quantifying what X-rays diagnostic prediction tasks generalize well across multiple different datasets. We present evidence that the issue of generalization is not due to a shift in the images but instead a shift in the labels. We study the cross-domain performance, agreement between models, and model representations. We find interesting discrepancies between performance and agreement where models which both achieve good performance disagree in their predictions as well as models which agree yet achieve poor performance. We also test for concept similarity by regularizing a network to group tasks across multiple datasets together and observe variation across the tasks. All code is made available online and data is publicly available: https://github.com/mlmed/torchxrayvision

Motivation & Objective

• Quantify how well X-ray diagnostic prediction tasks generalize across multiple public datasets
• Disentangle whether generalization issues arise from image shifts or label shifts
• Examine cross-domain performance, model agreement, and internal representations across datasets
• Assess whether training on combined datasets reflects true generalization or domain leakage
• Provide recommendations on evaluating and deploying cross-domain chest X-ray models

Proposed method

• Train an ensemble of DenseNets on four large public chest X-ray datasets using 200k+ images with AP/PA views
• Balance task losses by dataset-specific label frequencies to address label imbalance
• Calibrate model outputs per task using a dataset-specific optimal operating point to enable fair comparisons
• Evaluate cross-domain generalization via leave-one-domain-out and all-domains-inclusive experiments
• Analyze model agreement using Cohen’s kappa across models trained on different datasets
• Regularize task-specific weight vectors to study representational alignment across datasets

Experimental results

Research questions

• RQ1 Which chest X-ray prediction tasks generalize best across different datasets/institutions?
• RQ2 Is generalization limited by image distribution shifts (covariate shift) or by label distribution shifts (concept shift)?
• RQ3 How do model predictions, agreement between models, and internal representations vary across domains?
• RQ4 Can aligning representations across tasks/datasets improve cross-domain performance?
• RQ5 What are the implications for deploying cross-domain chest X-ray models in clinical settings?

Key findings

• Generalization performance varies by task and dataset; some tasks generalize better than others across domains
• Performance gaps exist even when a model trained on multiple domains is evaluated on a held-out domain, indicating domain-specific biases beyond simple covariate shift
• Models can show high agreement yet have poor predictive performance, and conversely, models with good performance may disagree substantially on predictions
• Calibrating outputs per task and dataset is essential to fair cross-domain evaluation and reveals the impact of label/ground-truth subjectivity
• Joint training on all datasets improves overall AUC but this reflects domain leakage rather than true generalization, as leave-one-domain-out tests show reduced benefits
• Representations diverge across datasets for the same task, and regularizing weight vectors to align task representations yields mixed success, highlighting intrinsic dataset-specific concept shifts