[Paper Review] Last-Layer Fairness Fine-tuning is Simple and Effective for Neural Networks

Yuzhen Mao, Zhun Deng+4 more|arXiv (Cornell University)|Apr 8, 2023

Welding Techniques and Residual Stresses4 citations

TL;DR

This paper proposes a simple yet effective method for training fair deep neural networks by fine-tuning only the last layer of a pre-trained model while applying fairness constraints and data reweighting. The approach significantly reduces overfitting to fairness criteria and achieves state-of-the-art fairness-accuracy trade-offs, outperforming full-network fine-tuning and other baselines on benchmark datasets.

ABSTRACT

As machine learning has been deployed ubiquitously across applications in modern data science, algorithmic fairness has become a great concern. Among them, imposing fairness constraints during learning, i.e. in-processing fair training, has been a popular type of training method because they don't require accessing sensitive attributes during test time in contrast to post-processing methods. While this has been extensively studied in classical machine learning models, their impact on deep neural networks remains unclear. Recent research has shown that adding fairness constraints to the objective function leads to severe over-fitting to fairness criteria in large models, and how to solve this challenge is an important open question. To tackle this, we leverage the wisdom and power of pre-training and fine-tuning and develop a simple but novel framework to train fair neural networks in an efficient and inexpensive way -- last-layer fine-tuning alone can effectively promote fairness in deep neural networks. This framework offers valuable insights into representation learning for training fair neural networks.

Motivation & Objective

• To address the challenge of fairness overfitting in over-parameterized deep neural networks during in-processing training.
• To explore whether fine-tuning only the last layer of a pre-trained model can effectively promote fairness without full-network retraining.
• To evaluate the effectiveness of combining data reweighting and fairness constraints in last-layer fine-tuning for improved fairness generalization.
• To investigate whether the proposed method generalizes to out-of-domain data and alternative fine-tuning structures.

Proposed method

• Pre-train a deep neural network using standard empirical risk minimization on a source dataset to learn robust representations.
• Fine-tune only the final classification layer using a fairness-constrained objective, while keeping all other layers frozen.
• Apply data reweighting during last-layer fine-tuning to balance minority and majority groups in the training data.
• Use stochastic gradient descent with momentum and weight decay to optimize the fairness-regularized objective on the last layer.
• Introduce a novel fairness-accuracy hybrid metric (AF) to jointly evaluate model performance across fairness and accuracy.
• Evaluate the method across multiple fairness notions: Equalized Odds Difference, Accuracy Equality Difference, and Worst Accuracy.

Experimental results

Research questions

• RQ1 Can last-layer fine-tuning with fairness constraints effectively mitigate overfitting to fairness objectives in large neural networks?
• RQ2 Does combining data reweighting with last-layer fine-tuning improve fairness generalization on imbalanced datasets?
• RQ3 How does the performance of last-layer fine-tuning compare to full-network fine-tuning with fairness constraints in terms of fairness and accuracy?
• RQ4 Can the proposed method generalize to out-of-domain data when the representation is trained on a source domain?
• RQ5 Does fine-tuning only the last layer preserve or even enhance model robustness compared to fine-tuning deeper layers?

Key findings

• FDR (Fine-tuning with Data Reweighting) achieved the highest fairness-accuracy (AF) score of 1.701 on the Worst Accuracy metric, outperforming all baselines.
• The FDR method reduced the Equalized Odds Difference (EO_Diff) on test data to 0.107, significantly lower than FullFT-Reg (0.499) and LastFT-Reg (0.153).
• Last-layer fine-tuning reduced the generalization gap between training and test fairness metrics, indicating reduced overfitting compared to full-network training.
• FDR achieved a test WACC of 0.892 and AUC of 0.958, demonstrating strong predictive performance while maintaining high fairness.
• The method generalized well to out-of-domain data, showing robustness under transfer learning settings.
• Fine-tuning only the last layer with fairness constraints yielded better fairness metrics than full-network fine-tuning, despite lower accuracy, due to reduced overfitting.