[论文解读] FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Models

Ruinan Jin, Zikang Xu+5 more|arXiv (Cornell University)|Jul 1, 2024

Health Systems, Economic Evaluations, Quality of Life被引用 4

一句话总结

FairMedFM 提出了一项全面的医疗影像基础模型（FMs）公平性基准，整合了涵盖多种模态和维度的17个多样化医疗数据集，并通过多种下游任务（分类与分割）及公平性感知技术评估了20个FMs。主要发现显示，各类模型中普遍存在偏见，效用-公平性权衡显著，现有缓解策略效果有限，且无一FMs在公平性或效用方面始终优于其他模型。

ABSTRACT

The advent of foundation models (FMs) in healthcare offers unprecedented opportunities to enhance medical diagnostics through automated classification and segmentation tasks. However, these models also raise significant concerns about their fairness, especially when applied to diverse and underrepresented populations in healthcare applications. Currently, there is a lack of comprehensive benchmarks, standardized pipelines, and easily adaptable libraries to evaluate and understand the fairness performance of FMs in medical imaging, leading to considerable challenges in formulating and implementing solutions that ensure equitable outcomes across diverse patient populations. To fill this gap, we introduce FairMedFM, a fairness benchmark for FM research in medical imaging.FairMedFM integrates with 17 popular medical imaging datasets, encompassing different modalities, dimensionalities, and sensitive attributes. It explores 20 widely used FMs, with various usages such as zero-shot learning, linear probing, parameter-efficient fine-tuning, and prompting in various downstream tasks -- classification and segmentation. Our exhaustive analysis evaluates the fairness performance over different evaluation metrics from multiple perspectives, revealing the existence of bias, varied utility-fairness trade-offs on different FMs, consistent disparities on the same datasets regardless FMs, and limited effectiveness of existing unfairness mitigation methods. Checkout FairMedFM's project page and open-sourced codebase, which supports extendible functionalities and applications as well as inclusive for studies on FMs in medical imaging over the long term.

研究动机与目标

• 解决医疗影像基础模型（FMs）在公平性评估方面缺乏标准化、全面基准的问题。
• 系统性评估不同FMs、数据模态、敏感属性（如性别、年龄、种族）以及下游任务（分类与分割）下的公平性表现。
• 探究不同微调策略与FMs类型下，模型效用与公平性之间的权衡关系。
• 提供可扩展的开源代码库，以支持医疗FMs公平性研究的长期发展。
• 评估现有公平性缓解技术在真实医疗影像场景中的有效性。

提出的方法

• 整合17个医疗影像数据集，涵盖2D、2.5D与3D模态（X光、CT、MRI、超声、OCT、眼底照相、皮肤镜）。
• 支持20种广泛使用的FMs，包括通用模型（如CLIP、ViT）与医疗专用模型（如Med-PaLM、SAM-Med3D）。
• 支持多种使用范式：零样本、线性探测、参数高效微调（PEFT）、基于提示的适配及全量微调。
• 采用基于PyTorch的统一代码库，模块化设计，包含数据加载器、模型、使用封装器、训练器与评估模块。
• 应用多样化的公平性度量指标，包括EqOdds、AUCΔ、ECEΔ、DSCΔ及预测对齐公平性，通过t-SNE分析表示公平性。
• 集成最先进的公平性缓解技术，如组重平衡、对抗训练与公平性约束。

Figure 1: Overview of the FairMedFM framework, a standardized pipeline to investigate fairness on diverse datasets (2D, 2.5D, and 3D), comprehensive functionalities (various FMs, tasks, usages, and debias algorithms), thorough evaluation metrics. The details are explained in Sec. 3 .

实验结果

研究问题

• RQ1 不同基础模型在多样化医疗影像数据集与敏感属性（如性别、年龄、种族）下的公平性表现如何？
• RQ2 在不同FMs类型、使用策略与下游任务下，效用-公平性权衡的性质与程度如何？
• RQ3 现有公平性缓解技术在多大程度上能减少医疗影像FMs中的偏见？
• RQ4 当应用于同一数据集时，不同FMs之间的公平性差异是否具有一致性？
• RQ5 FMs的潜在表征与公平性结果之间有何关联？能否通过表征空间操作缓解偏见？

主要发现

• 统计检验（CD图）确认，无一基础模型在公平性或效用方面始终优于其他模型，各模型间性能差异无显著性。
• 无论使用何种FMs，敏感属性（如性别、年龄、种族）之间均存在显著公平性差异，表明模型行为中存在持续偏见。
• 效用-公平性权衡显著，且在不同FMs类型与使用策略间表现各异，无单一方法能实现最优平衡。
• 现有公平性缓解方法（如对抗训练与组重平衡）在跨数据集与任务中减少偏见的效果有限。
• 表征空间分析显示，数据层面的组间可分性（如HAM10000中）与更高公平性差异相关，表明数据分布是偏见的关键来源。
• FairMedFM的开源代码库支持可扩展的基准测试，适用于多样化应用场景，包括基于提示的模型（如SAM-Med2D与FastSAM-3D）在分割任务中的应用。

Figure 2: Bias in classification tasks. $\text{AUC}_{\Delta}$ is the fairness evaluation metric. “ $\dagger$ ” denotes vision-language models, and “ $*$ ” denotes pure vision models, where CLIP-ZS and CLIP-Adapt are not applicable.