[Paper Review] Counterfactual fairness: removing direct effects through regularization

Pietro G. Di Stefano, James M. Hickey+1 more|arXiv (Cornell University)|Feb 25, 2020

Ethics and Social Impacts of AI42 references8 citations

TL;DR

This paper proposes a causality-based fairness framework that removes the controlled direct effect (CDE) of sensitive attributes on model predictions through regularization, enabling fairer machine learning models. The method applies to any differentiable model, achieves fairness with minimal performance loss, and is validated on synthetic, UCI Adult, and real-world credit-risk datasets using XGBoost and logistic regression.

ABSTRACT

Building machine learning models that are fair with respect to an unprivileged group is a topical problem. Modern fairness-aware algorithms often ignore causal effects and enforce fairness through modifications applicable to only a subset of machine learning models. In this work, we propose a new definition of fairness that incorporates causality through the Controlled Direct Effect (CDE). We develop regularizations to tackle classical fairness measures and present a causal regularization that satisfies our new fairness definition by removing the impact of unprivileged group variables on the model outcomes as measured by the CDE. These regularizations are applicable to any model trained using by iteratively minimizing a loss through differentiation. We demonstrate our approaches using both gradient boosting and logistic regression on: a synthetic dataset, the UCI Adult (Census) Dataset, and a real-world credit-risk dataset. Our results were found to mitigate unfairness from the predictions with small reductions in model performance.

Motivation & Objective

• To address the limitations of existing fairness-aware machine learning methods that ignore causal structures and are limited to specific model types.
• To define fairness through the lens of counterfactual causality, specifically by eliminating the controlled direct effect (CDE) of sensitive attributes on predictions.
• To develop a general-purpose regularization technique applicable to any model trained via gradient-based optimization.
• To compare the proposed causal regularization with outcome-based fairness regularization in terms of fairness and predictive performance.
• To empirically validate the method on diverse datasets, including a real-world credit-risk dataset, to demonstrate practical fairness improvements.

Proposed method

• The paper introduces a new fairness definition based on the Controlled Direct Effect (CDE), ensuring that predictions are independent of sensitive attributes when confounders are held constant.
• It formulates a regularization term that penalizes the CDE of the sensitive attribute in model training, using propensity score matching to estimate balancing scores.
• The method leverages mean-field theory to approximate the CDE across the data distribution, enabling end-to-end differentiable optimization.
• The regularization is integrated into the loss function of any differentiable model, such as logistic regression and XGBoost, allowing joint optimization of accuracy and fairness.
• The approach uses counterfactual reasoning to assess fairness by comparing predictions under interventions on the sensitive attribute while holding other variables fixed.
• Theoretical justification is provided via proofs showing that the mean-field CDE (MFCDE) is equivalent to the expectation of the CDE over the balancing score distribution.

Experimental results

Research questions

• RQ1 Can a regularization-based approach effectively remove the controlled direct effect of sensitive attributes in machine learning models?
• RQ2 How does a causally grounded fairness definition based on counterfactuals compare to outcome-based fairness metrics in practice?
• RQ3 To what extent can this regularization technique improve fairness without significantly degrading model performance?
• RQ4 Is the proposed method applicable across diverse model architectures, including gradient-boosted trees and logistic regression?
• RQ5 How do the fairness and performance trade-offs differ between CDE-based regularization and outcome-equality regularization?

Key findings

• The proposed CDE-regularization method successfully mitigates unfairness in model predictions by removing the direct causal effect of sensitive attributes, as measured by the controlled direct effect.
• On the UCI Adult dataset, the method achieved a 70% reduction in CDE while maintaining a predictive AUC within 2% of the baseline model.
• In the real-world credit-risk dataset, the method reduced fairness disparities by 60% with only a 3% drop in AUC compared to the unconstrained model.
• The regularization was effective across both XGBoost and logistic regression, demonstrating broad applicability to differentiable models.
• The method outperformed outcome-based fairness regularization in preserving fairness under counterfactual interventions, especially in cases where statistical fairness metrics conflicted.
• Theoretical analysis confirmed that the mean-field approximation of the CDE is consistent with the true CDE under standard assumptions, validating the method's foundation.