[Paper Review] Neither Private Nor Fair: Impact of Data Imbalance on Utility and Fairness in Differential Privacy

Tom Farrand, Fatemehsadat Mireshghallah+2 more|arXiv (Cornell University)|Sep 10, 2020

Privacy-Preserving Technologies in Data16 citations

TL;DR

This paper investigates how data imbalance and differential privacy (DP) jointly degrade model utility and fairness in deep learning. Using DP-SGD on imbalanced datasets, the authors show that even mild imbalance (e.g., 30% minority group) and loose privacy budgets (ε=4.98, ε=16.2) increase accuracy disparity between subgroups by 5.1–52%, with fairness metrics (demographic parity and equality of opportunity) worsening significantly under imbalance and DP.

ABSTRACT

Deployment of deep learning in different fields and industries is growing day by day due to its performance, which relies on the availability of data and compute. Data is often crowd-sourced and contains sensitive information about its contributors, which leaks into models that are trained on it. To achieve rigorous privacy guarantees, differentially private training mechanisms are used. However, it has recently been shown that differential privacy can exacerbate existing biases in the data and have disparate impacts on the accuracy of different subgroups of data. In this paper, we aim to study these effects within differentially private deep learning. Specifically, we aim to study how different levels of imbalance in the data affect the accuracy and the fairness of the decisions made by the model, given different levels of privacy. We demonstrate that even small imbalances and loose privacy guarantees can cause disparate impacts.

Motivation & Objective

• To investigate the combined impact of data imbalance and differential privacy on model accuracy and fairness in deep learning.
• To examine whether disparate impacts on minority subgroups occur even under loose privacy guarantees (low ε) and mild data imbalance.
• To analyze how gradient clipping in DP-SGD exacerbates accuracy gaps between majority and minority groups during training.
• To evaluate fairness metrics—demographic parity and equality of opportunity—under varying levels of data imbalance and privacy budgets.
• To identify conditions under which DP fails to ensure fair model performance despite rigorous privacy guarantees.

Proposed method

• Empirical evaluation of DP-SGD on image classification tasks (CelebA, UTKFace) with controlled data imbalance levels (0.1% to 30% minority group).
• Systematic variation of privacy budget ε (from 1.15 to 16.2) and δ=10⁻⁶ to assess impact on model accuracy and fairness.
• Use of moments accountant to track and enforce privacy budget throughout training.
• Monitoring of subgroup accuracy (e.g., female vs. male in CelebA) across training steps to observe divergence dynamics.
• Measurement of fairness metrics: difference in demographic parity (ΔDemP) and difference in equality of opportunity (ΔEO) across sensitive subgroups.
• Analysis of gradient clipping effects, hypothesizing that minority group examples are more likely to be clipped due to lower frequency in mini-batches.

Experimental results

Research questions

• RQ1 How does data imbalance affect the accuracy disparity between subgroups when using differentially private deep learning?
• RQ2 Does the disparate impact of DP on model accuracy persist even under loose privacy guarantees (e.g., ε=4.98 or ε=16.2)?
• RQ3 How do fairness metrics like demographic parity and equality of opportunity change as data imbalance increases under DP?
• RQ4 To what extent does gradient clipping in DP-SGD amplify accuracy gaps between majority and minority groups during training?
• RQ5 Can high privacy budgets (tight ε) mitigate fairness degradation, or do they instead reduce utility so much that fairness improves?

Key findings

• Even with a relatively balanced dataset (30% minority group), applying DP-SGD with ε=4.98 or ε=16.2 increases accuracy disparity between subgroups by 5.1% and 5.2%, respectively.
• When the minority group is reduced to 0.1% of the data, accuracy disparity increases to 19% and 52% under ε=4.98 and ε=16.2, respectively.
• Fairness metrics deteriorate significantly with increasing data imbalance: ΔDemP and ΔEO worsen across all privacy levels, with the most severe degradation in highly imbalanced settings.
• The divergence in subgroup accuracy is driven by gradient clipping, which disproportionately affects minority group examples due to their low frequency in mini-batches.
• Tighter privacy budgets (e.g., ε=1.15) can reduce disparity not by improving fairness, but by degrading overall utility so much that the model becomes more random and thus more balanced in predictions.
• The study demonstrates that DP does not ensure fairness and can exacerbate existing biases, even in datasets with negligible imbalance and loose privacy guarantees.