[Paper Review] Spending Privacy Budget Fairly and Wisely

Lucas Rosenblatt, Joshua E. Allen+1 more|arXiv (Cornell University)|Apr 27, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes SuperQUAIL, an ensemble method for differentially private synthetic data generation that allocates privacy budget wisely based on feature importance and fairly across subgroups to improve predictive accuracy and reduce disparities. It achieves higher overall and group-specific predictive performance than state-of-the-art methods, especially at moderate to high privacy budgets, while maintaining fairness in false negative rates across demographic groups.

ABSTRACT

Differentially private (DP) synthetic data generation is a practical method for improving access to data as a means to encourage productive partnerships. One issue inherent to DP is that the "privacy budget" is generally "spent" evenly across features in the data set. This leads to good statistical parity with the real data, but can undervalue the conditional probabilities and marginals that are critical for predictive quality of synthetic data. Further, loss of predictive quality may be non-uniform across the data set, with subsets that correspond to minority groups potentially suffering a higher loss. In this paper, we develop ensemble methods that distribute the privacy budget "wisely" to maximize predictive accuracy of models trained on DP data, and "fairly" to bound potential disparities in accuracy across groups and reduce inequality. Our methods are based on the insights that feature importance can inform how privacy budget is allocated, and, further, that per-group feature importance and fairness-related performance objectives can be incorporated in the allocation. These insights make our methods tunable to social contexts, allowing data owners to produce balanced synthetic data for predictive analysis.

Motivation & Objective

• To address the limitation of uniform privacy budget allocation in differentially private synthetic data generation, which undermines predictive quality, especially for minority groups.
• To develop a method that allocates privacy budget based on feature importance to enhance predictive fidelity of synthetic data.
• To incorporate fairness objectives into privacy budget allocation, minimizing disparities in model performance across sub-populations.
• To enable data owners to produce balanced, interpretable, and tunable synthetic data that reflect social and ethical priorities.
• To demonstrate that privacy budget allocation can be optimized not only for utility but also for fairness in predictive modeling.

Proposed method

• The method uses an ensemble framework where a differentially private model (DPSAGE) estimates feature importance to guide privacy budget allocation.
• The privacy budget is split between a marginal-based synthesizer (e.g., MST) and a conditional model (e.g., DPLogisticRegression), with allocation tuned via grid search on epsilon splits.
• Fairness is enforced by optimizing for balanced performance metrics (e.g., FNR, FPR) across subgroups using a fairness-aware objective function.
• The framework includes two variants: FSQ-Bal for balanced accuracy and FSQ-FNR for minimizing false negative rates, particularly in high-impact scenarios.
• The method dynamically allocates epsilon between components based on feature importance and fairness constraints, enabling tunable trade-offs.
• Performance is evaluated using end-to-end classification tasks on real and synthetic data, with metrics including accuracy, F1 score, FNR, and FPR across demographic groups.

Experimental results

Research questions

• RQ1 Can privacy budget allocation based on feature importance improve the predictive accuracy of models trained on differentially private synthetic data?
• RQ2 Can fairness constraints be effectively integrated into the privacy budget allocation process to reduce performance disparities across demographic subgroups?
• RQ3 How does the performance of the proposed method compare to state-of-the-art DP synthesizers in terms of predictive utility and fairness at varying privacy budgets?
• RQ4 Does the method maintain high statistical fidelity while improving predictive and fairness metrics?
• RQ5 Can the framework be tuned to prioritize specific fairness objectives, such as minimizing false negative rates in underrepresented groups?

Key findings

• At ε = e³, FSQ-Bal achieved 72.0% overall accuracy on the ACSEmployment dataset, outperforming MST (65.2%) and approaching real data (74.3%).
• FSQ-Bal reduced the false negative rate (FNR) to 9% overall and 9% for the Black group, outperforming real data (12% overall, 16% for Black) and MST (10% overall, 21% for Black).
• The method achieved balanced FNR and FPR across racial groups, demonstrating effective fairness enforcement in synthetic data generation.
• At low privacy budgets (ε = e⁰), the method showed higher variance than MST, but surpassed MST in performance at ε = e² and ε = e³, indicating effectiveness at moderate-to-high privacy levels.
• Feature importance estimation via DPSAGE achieved strong alignment with real data (nDCG ≈ 0.9 at ε = 0.2), validating its use in budget allocation.
• The framework outperformed MST in predictive metrics (accuracy, F1) while maintaining fairness, even though MST had better aggregate statistical fidelity.