[Paper Review] Private, fair and accurate: Training large-scale, privacy-preserving AI models in medical imaging

Soroosh Tayebi Arasteh, Alexander Ziller+7 more|arXiv (Cornell University)|Feb 3, 2023

Artificial Intelligence in Healthcare and Education57 references12 citations

TL;DR

The paper evaluates privacy-preserving (differential privacy) training for medical imaging models, comparing accuracy and fairness to non-private training on chest radiographs and 3D CT PDAC tasks, finding feasible accuracy and fairness with some accuracy trade-offs.

ABSTRACT

Artificial intelligence (AI) models are increasingly used in the medical domain. However, as medical data is highly sensitive, special precautions to ensure its protection are required. The gold standard for privacy preservation is the introduction of differential privacy (DP) to model training. Prior work indicates that DP has negative implications on model accuracy and fairness, which are unacceptable in medicine and represent a main barrier to the widespread use of privacy-preserving techniques. In this work, we evaluated the effect of privacy-preserving training of AI models regarding accuracy and fairness compared to non-private training. For this, we used two datasets: (1) A large dataset (N=193,311) of high quality clinical chest radiographs, and (2) a dataset (N=1,625) of 3D abdominal computed tomography (CT) images, with the task of classifying the presence of pancreatic ductal adenocarcinoma (PDAC). Both were retrospectively collected and manually labeled by experienced radiologists. We then compared non-private deep convolutional neural networks (CNNs) and privacy-preserving (DP) models with respect to privacy-utility trade-offs measured as area under the receiver-operator-characteristic curve (AUROC), and privacy-fairness trade-offs, measured as Pearson's r or Statistical Parity Difference. We found that, while the privacy-preserving trainings yielded lower accuracy, they did largely not amplify discrimination against age, sex or co-morbidity. Our study shows that -- under the challenging realistic circumstances of a real-life clinical dataset -- the privacy-preserving training of diagnostic deep learning models is possible with excellent diagnostic accuracy and fairness.

Motivation & Objective

• Motivate privacy preservation for medical imaging AI due to sensitive data.
• Assess the impact of differential privacy on model accuracy (AUROC) and fairness metrics.
• Compare privacy-preserving models to non-private baselines on large clinical datasets.
• Characterize privacy-utility and privacy-fairness trade-offs in real-world data.

Proposed method

• Use two datasets: 193,311 chest radiographs and 1,625 3D abdominal CTs labeled for pancreatic ductal adenocarcinoma.
• Train deep CNNs with privacy-preserving (DP) methods and non-private baselines.
• Evaluate performance with AUROC and fairness measures (Pearson's r, Statistical Parity Difference).
• Analyze whether DP training degrades accuracy and whether it amplifies demographic disparities.
• Report results under realistic clinical data conditions.

Experimental results

Research questions

• RQ1 How does differential privacy affect diagnostic accuracy (AUROC) in large-scale medical imaging tasks?
• RQ2 Does privacy-preserving training amplify or mitigate discrimination across age, sex, or comorbidity factors?
• RQ3 Are privacy-preserving models feasible in real-life clinical datasets without severely compromising fairness and accuracy?

Key findings

• DP training yields lower accuracy than non-private models.
• DP training largely does not amplify discrimination against age, sex, or co-morbidity.
• Demonstrates that privacy-preserving diagnostic DL models can achieve excellent diagnostic accuracy and fairness in real-life clinical datasets.