[Paper Review] FairBatch: Batch Selection for Model Fairness

Yuji Roh, Kangwook Lee+2 more|arXiv (Cornell University)|Dec 3, 2020

Privacy-Preserving Technologies in Data46 references35 citations

TL;DR

FairBatch introduces an outer bilevel optimization that adaptively reweights minibatch sampling to improve group fairness (equal opportunity, equalized odds, demographic parity) without changing data preprocessing or inner model training. It can also fine-tune pre-trained models for fairness.

ABSTRACT

Training a fair machine learning model is essential to prevent demographic disparity. Existing techniques for improving model fairness require broad changes in either data preprocessing or model training, rendering themselves difficult-to-adopt for potentially already complex machine learning systems. We address this problem via the lens of bilevel optimization. While keeping the standard training algorithm as an inner optimizer, we incorporate an outer optimizer so as to equip the inner problem with an additional functionality: Adaptively selecting minibatch sizes for the purpose of improving model fairness. Our batch selection algorithm, which we call FairBatch, implements this optimization and supports prominent fairness measures: equal opportunity, equalized odds, and demographic parity. FairBatch comes with a significant implementation benefit -- it does not require any modification to data preprocessing or model training. For instance, a single-line change of PyTorch code for replacing batch selection part of model training suffices to employ FairBatch. Our experiments conducted both on synthetic and benchmark real data demonstrate that FairBatch can provide such functionalities while achieving comparable (or even greater) performances against the state of the arts. Furthermore, FairBatch can readily improve fairness of any pre-trained model simply via fine-tuning. It is also compatible with existing batch selection techniques intended for different purposes, such as faster convergence, thus gracefully achieving multiple purposes.

Motivation & Objective

• Motivate and enable easier integration of fairness in ML by modifying batch selection rather than data or model structure.
• Develop a bilevel optimization framework that treats batch sampling as an outer optimizer influencing inner model training.
• Provide a practical, minimally invasive FairBatch implementation compatible with common frameworks (e.g., PyTorch).
• Demonstrate FairBatch’s effectiveness across multiple fairness notions (EO, ED, DP) on synthetic and real datasets.
• Show that FairBatch can improve fairness of pre-trained models via fine-tuning and can coexist with other batch-selection techniques.

Proposed method

• Formulate batch selection as an outer optimization in a bilevel framework where the inner problem is standard SGD training.
• Define fairness-oriented outer objectives that minimize disparities such as |L_{y,z}(w_λ) − L_{y,⊥}(w_λ)| or a max-disparity objective across groups.
• Introduce one- or multi-dimensional outer variables λ that control the sampling probabilities for different sensitive-group strata.
• Provide update rules for λ that are simple (gradient-like) and provably converge in the one-dimensional case (d=1).
• Extend the outer updates to multi-dimensional settings (d>1) by using a subgradient approximation to avoid costly Hessian computations.
• Show compatibility with existing batch-selection methods by combining with importance-weighting schemes to accelerate convergence.

Experimental results

Research questions

• RQ1 Can adaptive batch selection, framed as a bilevel optimization, enforce common group fairness criteria (equal opportunity, equalized odds, demographic parity) during standard SGD training?
• RQ2 How should the outer optimization variables (sampling probabilities) be updated to reduce group disparities without sacrificing overall accuracy?
• RQ3 Is FairBatch effective across synthetic and real-world datasets and compatible with pre-trained models or other batch-selection techniques?
• RQ4 What are the theoretical guarantees (e.g., quasi-convexity and convergence) for the proposed one-dimensional outer optimization, and how does the multi-dimensional case perform in practice?
• RQ5 How easily can FairBatch be integrated into existing ML pipelines with minimal code changes?

Key findings

• FairBatch achieves comparable or better fairness (EO, ED, DP) compared with state-of-the-art fairness methods while preserving accuracy across synthetic and real datasets.
• On synthetic data, COMPAS, and AdultCensus, FairBatch reduces fairness disparities with similar or faster convergence (fewer epochs) than baselines.
• FairBatch can improve fairness of pre-trained models via fine-tuning (e.g., UTKFace with ResNet18/GoogLeNet) without sacrificing accuracy unlike some baselines.
• FairBatch is easy to implement in practice, requiring only a small change to the batch sampler in PyTorch (single-line modification).
• The method can be combined with other batch-selection strategies such as loss-based weighting to accelerate convergence while maintaining fairness benefits.
• Compared to AdaFair, FairBatch achieves similar fairness improvements but with a single-model training, offering substantial speed advantages (22.5–96x faster in reported experiments).