[Paper Review] Assessing Fairness in the Presence of Missing Data

Yiliang Zhang, Qi Long|PubMed|Dec 7, 2021

Privacy-Preserving Technologies in Data6 references11 citations

TL;DR

This paper proposes a re-weighting approach to estimate fairness in machine learning models trained on complete cases when data are missing, showing that proper weighting can reduce bias in fairness estimation. It provides the first theoretical bounds on fairness estimation error under missing data, demonstrating that estimation bias depends on the discrepancy between complete case and complete data distributions and the accuracy of propensity score weights.

ABSTRACT

Missing data are prevalent and present daunting challenges in real data analysis. While there is a growing body of literature on fairness in analysis of fully observed data, there has been little theoretical work on investigating fairness in analysis of incomplete data. In practice, a popular analytical approach for dealing with missing data is to use only the set of complete cases, i.e., observations with all features fully observed to train a prediction algorithm. However, depending on the missing data mechanism, the distribution of complete cases and the distribution of the complete data may be substantially different. When the goal is to develop a fair algorithm in the complete data domain where there are no missing values, an algorithm that is fair in the complete case domain may show disproportionate bias towards some marginalized groups in the complete data domain. To fill this significant gap, we study the problem of estimating fairness in the complete data domain for an arbitrary model evaluated merely using complete cases. We provide upper and lower bounds on the fairness estimation error and conduct numerical experiments to assess our theoretical results. Our work provides the first known theoretical results on fairness guarantee in analysis of incomplete data.

Motivation & Objective

• To address the critical gap in theoretical understanding of fairness estimation when data contain missing values.
• To investigate how missing data mechanisms (MCAR, MAR, MNAR) affect fairness estimation in complete case analysis.
• To develop a re-weighting method that corrects for bias in fairness estimation due to domain shift between complete cases and the target complete data population.
• To provide theoretical bounds on fairness estimation error, quantifying the impact of sample size and weighting accuracy.
• To validate the method empirically on real-world datasets (COMPAS and ADNI) under various missing data mechanisms.

Proposed method

• Proposes a re-weighting technique where complete cases are assigned weights based on estimated propensity scores to correct for selection bias in complete case analysis.
• Derives upper and lower bounds on fairness estimation error, decomposing it into terms related to sample size, total variation distance between distributions, and weighting accuracy.
• Uses the total variation distance between the complete case and complete data distributions as a key measure of domain shift.
• Applies the method to both classification and regression tasks using accuracy parity gap as the fairness metric.
• Employs logistic regression and other models to estimate missingness mechanisms (propensity scores) under MAR and MNAR settings.
• Validates theoretical results through simulations and real data experiments on COMPAS and ADNI datasets under MCAR, MAR, and MNAR mechanisms.

Experimental results

Research questions

• RQ1 How does missing data affect fairness estimation when only complete cases are used for model training?
• RQ2 What is the theoretical relationship between fairness estimation error and the discrepancy between complete case and complete data distributions?
• RQ3 Can re-weighting based on estimated propensity scores reduce fairness estimation bias in incomplete data settings?
• RQ4 How do different missing data mechanisms (MCAR, MAR, MNAR) impact the performance of fairness estimation under complete case analysis?
• RQ5 To what extent does sample imbalance between sensitive groups affect fairness estimation bias in incomplete data?

Key findings

• Under MCAR, all weighting strategies, including unweighted estimation, yield comparable and valid fairness estimates, as expected.
• Under MAR, the true propensity score weights yield the lowest bias, and logistic regression-based weights (correctly specified model) perform nearly as well.
• Under MNAR, all working models are mis-specified, and all weighting strategies yield larger bias than the true weights, highlighting the challenge of model misspecification.
• Sample imbalance between sensitive groups significantly increases fairness estimation bias, especially under non-ignorable missingness mechanisms.
• The theoretical bounds on fairness estimation error are validated empirically, showing that bias increases with domain shift and poor weighting accuracy.
• The results demonstrate that fairness estimation using complete cases can be substantially biased if the missing data mechanism is not properly accounted for, even when models are trained on complete cases.