[Paper Review] Explainable, Domain-Adaptive, and Federated Artificial Intelligence in Medicine

Ahmad Chaddad, Qizong lu+6 more|arXiv (Cornell University)|Nov 17, 2022

Radiomics and Machine Learning in Medical Imaging4 citations

TL;DR

This paper proposes a unified framework integrating explainable AI, domain adaptation, and federated learning to address key challenges in medical AI: model interpretability, domain shift across imaging systems, and data privacy. By combining post-hoc explainability, domain-invariant representation learning, and decentralized model training, the approach enables trustworthy, generalizable, and privacy-preserving AI in clinical settings.

ABSTRACT

Artificial intelligence (AI) continues to transform data analysis in many domains. Progress in each domain is driven by a growing body of annotated data, increased computational resources, and technological innovations. In medicine, the sensitivity of the data, the complexity of the tasks, the potentially high stakes, and a requirement of accountability give rise to a particular set of challenges. In this review, we focus on three key methodological approaches that address some of the particular challenges in AI-driven medical decision making. (1) Explainable AI aims to produce a human-interpretable justification for each output. Such models increase confidence if the results appear plausible and match the clinicians expectations. However, the absence of a plausible explanation does not imply an inaccurate model. Especially in highly non-linear, complex models that are tuned to maximize accuracy, such interpretable representations only reflect a small portion of the justification. (2) Domain adaptation and transfer learning enable AI models to be trained and applied across multiple domains. For example, a classification task based on images acquired on different acquisition hardware. (3) Federated learning enables learning large-scale models without exposing sensitive personal health information. Unlike centralized AI learning, where the centralized learning machine has access to the entire training data, the federated learning process iteratively updates models across multiple sites by exchanging only parameter updates, not personal health data. This narrative review covers the basic concepts, highlights relevant corner-stone and state-of-the-art research in the field, and discusses perspectives.

Motivation & Objective

• Address the clinical mistrust of black-box AI models by enhancing model interpretability through explainable AI (XAI) techniques.
• Overcome performance degradation due to domain shift in medical imaging caused by differences in scanners, protocols, and institutions.
• Enable collaborative model training across multiple healthcare institutions without sharing sensitive patient data, ensuring compliance with data privacy regulations.
• Integrate XAI, domain adaptation, and federated learning into a cohesive pipeline to support real-world clinical deployment of AI systems.
• Improve the reliability and generalization of deep learning models in heterogeneous, multi-institutional medical imaging environments.

Proposed method

• Employ post-hoc explainability methods such as class activation mapping (CAM) to generate saliency maps that highlight regions of medical images influencing model predictions.
• Apply feature-based domain adaptation techniques, including adversarial training, to align the feature distributions between source (training) and target (deployment) domains.
• Implement horizontal federated learning to train a global model across multiple institutions by aggregating local model updates while keeping raw patient data localized.
• Use a centralized aggregation server to iteratively update the global model using only model parameters (not raw data), preserving data privacy.
• Integrate domain adaptation within the federated learning pipeline to reduce domain shift across institutions with varying imaging protocols and equipment.
• Combine XAI with federated and domain-adaptive models to produce interpretable, generalizable, and privacy-preserving predictions in clinical decision support.

Experimental results

Research questions

• RQ1 How can explainable AI techniques improve clinician trust in deep learning models used for medical image analysis?
• RQ2 To what extent can domain adaptation techniques reduce performance degradation when deploying a medical AI model across different imaging sites with varying acquisition protocols?
• RQ3 Can federated learning enable effective model training across multiple healthcare institutions without exposing sensitive patient data?
• RQ4 What are the key challenges in combining XAI, domain adaptation, and federated learning for clinical AI applications?
• RQ5 How can these three methodologies be integrated into a single pipeline to support robust, generalizable, and privacy-preserving AI in medicine?

Key findings

• Post-hoc explainability methods like CAM provide local interpretability by highlighting image regions influencing predictions, but they do not reveal the internal decision logic of complex models.
• Adversarial domain adaptation methods show strong performance in reducing domain shift and improving model generalization across different imaging domains.
• Federated learning enables collaborative model training across institutions without sharing raw data, preserving patient privacy and complying with data protection regulations.
• Despite progress, federated learning in healthcare remains underutilized, with most studies focusing on horizontal FL and limited performance compared to centralized training.
• Data heterogeneity, label inconsistency, and poor data quality across institutions remain major challenges that hinder the performance and reliability of federated and domain-adaptive models.
• A unified framework combining XAI, domain adaptation, and federated learning is essential for building trustworthy, generalizable, and privacy-preserving AI systems in clinical practice.