[Paper Review] FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysis

Raman Dutt, Ondrej Bohdal+2 more|arXiv (Cornell University)|Oct 8, 2023

Artificial Intelligence in Healthcare and EducationMedicine3 citations

TL;DR

FairTune proposes a bi-level optimization framework to enhance fairness in medical image analysis by selecting optimal parameter-efficient fine-tuning (PEFT) masks that maximize validation fairness. By targeting the fairness generalization gap—where bias emerges during testing despite perfect training fairness—it outperforms standard empirical risk minimization (ERM) across diverse medical imaging benchmarks.

ABSTRACT

Training models with robust group fairness properties is crucial in ethically sensitive application areas such as medical diagnosis. Despite the growing body of work aiming to minimise demographic bias in AI, this problem remains challenging. A key reason for this challenge is the fairness generalisation gap: High-capacity deep learning models can fit all training data nearly perfectly, and thus also exhibit perfect fairness during training. In this case, bias emerges only during testing when generalisation performance differs across subgroups. This motivates us to take a bi-level optimisation perspective on fair learning: Optimising the learning strategy based on validation fairness. Specifically, we consider the highly effective workflow of adapting pre-trained models to downstream medical imaging tasks using parameter-efficient fine-tuning (PEFT) techniques. There is a trade-off between updating more parameters, enabling a better fit to the task of interest vs. fewer parameters, potentially reducing the generalisation gap. To manage this tradeoff, we propose FairTune, a framework to optimise the choice of PEFT parameters with respect to fairness. We demonstrate empirically that FairTune leads to improved fairness on a range of medical imaging datasets. The code is available at https://github.com/Raman1121/FairTune

Motivation & Objective

• Address the fairness generalization gap in medical AI, where models exhibit bias during testing despite perfect fairness during training.
• Overcome the ineffectiveness of existing fairness interventions that rely on training-time constraints, which fail on high-capacity deep models.
• Develop a method to optimize PEFT by selecting the most fairness-promoting parameter update masks during fine-tuning.
• Demonstrate that fairness can be improved by regularizing learning based on validation performance rather than training performance.
• Provide a scalable, efficient framework applicable to real-world medical imaging tasks using pre-trained models.

Proposed method

• Formulate fair learning as a bi-level optimization problem: optimize the PEFT update mask to maximize fairness on validation data.
• Use parameter-efficient fine-tuning (PEFT) to constrain the number of updated parameters, reducing overfitting and generalization gaps.
• Apply a differentiable search mechanism to identify the optimal parameter update mask that minimizes fairness disparities on validation data.
• Train the model using standard ERM loss but with the mask constrained to only update a subset of parameters, selected via fairness-driven optimization.
• Perform hyperparameter search on a representative subset of data to reduce computational cost while preserving fairness performance.
• Evaluate fairness using metrics like equalized odds and equal opportunity across protected attributes (e.g., age, gender, skin type).

Figure 1: Bias arises during train-test generalisation. Left (Training AUROC): High-capacity deep models can exhibit perfect group fairness during training because they can classify all the training data perfectly. Right (Validation AUROC): Bias arises because the disadvantaged subgroup has worse ge

Experimental results

Research questions

• RQ1 Can fairness be improved in medical image analysis by optimizing the PEFT update mask based on validation fairness rather than training fairness?
• RQ2 Does the fairness generalization gap—where bias emerges during testing despite perfect training fairness—persist in high-capacity models?
• RQ3 Can bi-level optimization of PEFT parameters lead to consistent fairness improvements across diverse medical imaging datasets?
• RQ4 How does FairTune compare to standard ERM fine-tuning in terms of fairness metrics across multiple protected attributes?
• RQ5 Is the proposed method scalable and effective when applied to large-scale datasets like CheXpert?

Key findings

• FairTune consistently improves fairness metrics—such as equalized odds and equal opportunity—across all evaluated medical imaging datasets, including Fitzpatrick17k, HAM10000, PAPILA, OL3I, OASIS, Harvard-GF3300, and CheXpert.
• The method outperforms standard ERM fine-tuning in fairness, even when existing fairness interventions fail due to the fairness generalization gap.
• Empirical results confirm that bias emerges during test-time generalization, not during training, validating the core motivation of the paper.
• The use of a subset for hyperparameter search (e.g., 10% of CheXpert) maintains fairness performance on the full dataset, demonstrating scalability.
• FairTune achieves statistically significant fairness improvements over ERM without compromising overall accuracy, indicating effective bias mitigation.
• The framework is robust across diverse modalities (dermatology, CT, MRI, X-ray) and sensitive attributes (age, gender, skin type, race).

Figure 2: Illustration that shows how our approach optimises the structure of PEFT with respect to fairness. Hyperparameter optimisation (HPO) selects a mask that decides which components of a pre-trained model $\theta$ are fine-tuned using PEFT. For each sampled mask, the fine-tuned model is evalua