[Paper Review] Representative & Fair Synthetic Data

Paul Tiwald, Alexandra Ebert+1 more|arXiv (Cornell University)|Apr 7, 2021

Ethics and Social Impacts of AI7 references7 citations

TL;DR

This paper proposes a framework to generate representative and fair synthetic tabular data by integrating fairness constraints—specifically statistical parity—into the self-supervised training of generative models. By adding a fairness loss to the model's objective function, the approach mitigates gender and racial biases in the UCI Adult dataset while preserving statistical relationships, resulting in downstream models that assign nearly equal propensity scores to protected groups without explicit fairness optimization.

ABSTRACT

Algorithms learn rules and associations based on the training data that they are exposed to. Yet, the very same data that teaches machines to understand and predict the world, contains societal and historic biases, resulting in biased algorithms with the risk of further amplifying these once put into use for decision support. Synthetic data, on the other hand, emerges with the promise to provide an unlimited amount of representative, realistic training samples, that can be shared further without disclosing the privacy of individual subjects. We present a framework to incorporate fairness constraints into the self-supervised learning process, that allows to then simulate an unlimited amount of representative as well as fair synthetic data. This framework provides a handle to govern and control for privacy as well as for bias within AI at its very source: the training data. We demonstrate the proposed approach by amending an existing generative model architecture and generating a representative as well as fair version of the UCI Adult census data set. While the relationships between attributes are faithfully retained, the gender and racial biases inherent in the original data are controlled for. This is further validated by comparing propensity scores of downstream predictive models that are trained on the original data versus the fair synthetic data. We consider representative & fair synthetic data a promising future building block to teach algorithms not on historic worlds, but rather on the worlds that we strive to live in.

Motivation & Objective

• To address systemic biases in machine learning by controlling for fairness at the data generation stage.
• To develop a method that generates synthetic data that is both representative of the original data and fair with respect to protected attributes like gender and race.
• To demonstrate that fairness can be embedded directly into the data generation process, reducing bias in downstream models without modifying the learning algorithm.
• To validate that fairness constraints in synthetic data generation effectively reduce proxy bias and maintain data quality.

Proposed method

• Integrate a fairness loss into the training objective of a generative model, alongside the standard accuracy loss, to optimize for both data fidelity and fairness.
• Use statistical parity as the fairness definition, where the probability of a positive outcome (e.g., high income) is equal across protected groups (e.g., male vs. female).
• Formulate the fairness loss as a penalty proportional to the deviation from empirical statistical parity, enabling a trade-off between accuracy and fairness via weighted loss combination.
• Train the model end-to-end on the Adult dataset with gender and race as protected attributes, applying fairness constraints during synthesis.
• Preserve univariate and bivariate statistical distributions of the original data to ensure representativeness of the synthetic samples.
• Evaluate fairness by analyzing propensity scores of downstream logistic regression models trained on synthetic vs. original data.

Experimental results

Research questions

• RQ1 Can fairness constraints be effectively embedded into the data generation process to produce synthetic data that is both representative and fair?
• RQ2 Does training generative models with a fairness loss reduce bias in downstream predictive models without explicit fairness optimization?
• RQ3 To what extent can fairness constraints account for proxy variables that encode sensitive attributes indirectly?
• RQ4 How does the inclusion of multiple fairness constraints (e.g., for both gender and race) affect the balance of outcomes across intersectional groups?

Key findings

• The synthetic data achieved a disparate impact ratio of 0.88 for gender, exceeding the 0.8 threshold (four-fifths rule), compared to 0.33 in the original data, indicating significant bias mitigation.
• Univariate and bivariate statistical distributions of the original data were preserved in the synthetic data, confirming high representativeness.
• The correlation between gender and proxy attributes was maintained, but the correlation between proxy attributes and income was significantly reduced, demonstrating effective mitigation of proxy bias.
• Downstream logistic regression models trained on the synthetic data assigned nearly equal propensity scores to men and women, even though no fairness constraints were applied during model training.
• The synthetic models achieved competitive performance (84.13% accuracy, 88.34% AUC), showing that fairness does not come at the cost of predictive performance.
• When fairness constraints were applied simultaneously to race and gender, the high-income ratios across all four intersectional groups (e.g., White male, Black female) were significantly balanced, with further improvement possible by increasing the fairness loss weight.