[Paper Review] Embracing the Disharmony in Medical Imaging: A Simple and Effective Framework for Domain Adaptation

Rongguang Wang, Pratik Chaudhari+1 more|arXiv (Cornell University)|Mar 23, 2021

Domain Adaptation and Few-Shot LearningComputer Science111 references58 citations

TL;DR

This paper proposes a novel domain adaptation framework that embraces, rather than corrects, data disharmony in medical imaging by leveraging fine-tuning with label-aware adaptation and auxiliary tasks. It achieves significant performance gains in both intra-study adaptation and inter-study generalization on 3D brain MRI datasets for Alzheimer’s disease and schizophrenia classification, outperforming methods that enforce representation invariance.

ABSTRACT

Domain shift, the mismatch between training and testing data characteristics, causes significant degradation in the predictive performance in multi-source imaging scenarios. In medical imaging, the heterogeneity of population, scanners and acquisition protocols at different sites presents a significant domain shift challenge and has limited the widespread clinical adoption of machine learning models. Harmonization methods which aim to learn a representation of data invariant to these differences are the prevalent tools to address domain shift, but they typically result in degradation of predictive accuracy. This paper takes a different perspective of the problem: we embrace this disharmony in data and design a simple but effective framework for tackling domain shift. The key idea, based on our theoretical arguments, is to build a pretrained classifier on the source data and adapt this model to new data. The classifier can be fine-tuned for intra-site domain adaptation. We can also tackle situations where we do not have access to ground-truth labels on target data; we show how one can use auxiliary tasks for adaptation; these tasks employ covariates such as age, gender and race which are easy to obtain but nevertheless correlated to the main task. We demonstrate substantial improvements in both intra-site domain adaptation and inter-site domain generalization on large-scale real-world 3D brain MRI datasets for classifying Alzheimer's disease and schizophrenia.

Motivation & Objective

• Address the challenge of domain shift in multi-site medical imaging, where scanner, population, and protocol differences degrade model generalization.
• Overcome the limitations of existing harmonization methods that reduce predictive accuracy by enforcing representation invariance.
• Develop a practical framework for intra-study adaptation using label information and for inter-study generalization without target labels.
• Utilize easily accessible demographic covariates (age, gender, race) as auxiliary tasks to guide model adaptation to new imaging conditions.
• Improve model performance across diverse age groups, racial populations, and scanner types, particularly for underrepresented subgroups.

Proposed method

• Pre-train a deep classifier on source data and fine-tune it on subgroups (e.g., age, race, scanner type) using label information for intra-study adaptation.
• Introduce a regularization term in the fine-tuning process to stabilize adaptation across diverse populations and imaging devices.
• For inter-study generalization, use auxiliary tasks—such as sex classification and age regression—on target-domain data without requiring ground-truth labels.
• Train a shared feature extractor for both the primary task (e.g., disease classification) and auxiliary tasks using source data.
• Fine-tune the feature extractor on the target domain using only auxiliary task supervision, while regularizing the primary task head using source data.
• Leverage demographic covariates as proxy signals to align features across domains, enabling transferability without direct label access.

Experimental results

Research questions

• RQ1 Can embracing domain disharmony through fine-tuning improve predictive accuracy compared to learning invariant representations in medical imaging?
• RQ2 How effective is auxiliary task learning (e.g., age and sex prediction) in enabling domain generalization when target labels are unavailable?
• RQ3 To what extent can label-aware fine-tuning on subgroups (e.g., age ranges, scanner types) improve model performance within a single study?
• RQ4 Does age prediction serve as a more effective auxiliary task than sex classification for cross-study transfer in neuroimaging?
• RQ5 Can transfer learning mitigate performance disparities in underrepresented groups (e.g., older adults, African Americans) across imaging studies?

Key findings

• The proposed method improved Alzheimer’s disease classification accuracy from 88.62% to 97.10% in the >80 age group, demonstrating strong intra-study adaptation gains.
• For African-American participants in the PENN dataset, prediction accuracy increased from 95.36% to 98.15% after transfer learning, reducing disparities.
• Caucasian group accuracy rose from 90.23% to 97.93% in the PENN dataset, showing benefits even for majority populations.
• In the AIBL dataset, Siemens Trio scanner performance improved from 90.88% to 98.51% after adaptation, highlighting scanner-specific gains.
• Age prediction consistently outperformed sex classification as an auxiliary task across all experiments, suggesting its stronger alignment with neuroimaging domain shifts.
• The framework achieved substantial improvements in inter-study generalization, with schizophrenia classification accuracy rising from 51.02% (baseline) to higher levels when using auxiliary tasks.