[Paper Review] Fairness-Accuracy Trade-Offs: A Causal Perspective

Drago Plečko, Elias Bareinboim|arXiv (Cornell University)|May 24, 2024

Climate Change Policy and EconomicsEconomics, Econometrics and Finance3 citations

TL;DR

This paper introduces a causal framework to analyze fairness-accuracy trade-offs in machine learning by proposing path-specific excess loss (PSEL) to quantify the accuracy cost of enforcing fairness along individual causal pathways. It decomposes total excess loss into local PSELs, introduces the causal fairness/utility ratio for comparing trade-offs across pathways, and develops a neural method for causally constrained fair learning, demonstrating empirically that fairness and accuracy are consistently in tension across real-world datasets.

ABSTRACT

Systems based on machine learning may exhibit discriminatory behavior based on sensitive characteristics such as gender, sex, religion, or race. In light of this, various notions of fairness and methods to quantify discrimination were proposed, leading to the development of numerous approaches for constructing fair predictors. At the same time, imposing fairness constraints may decrease the utility of the decision-maker, highlighting a tension between fairness and utility. This tension is also recognized in legal frameworks, for instance in the disparate impact doctrine of Title VII of the Civil Rights Act of 1964 -- in which specific attention is given to considerations of business necessity -- possibly allowing the usage of proxy variables associated with the sensitive attribute in case a high-enough utility cannot be achieved without them. In this work, we analyze the tension between fairness and accuracy from a causal lens for the first time. We introduce the notion of a path-specific excess loss (PSEL) that captures how much the predictor's loss increases when a causal fairness constraint is enforced. We then show that the total excess loss (TEL), defined as the difference between the loss of predictor fair along all causal pathways vs. an unconstrained predictor, can be decomposed into a sum of more local PSELs. At the same time, enforcing a causal constraint often reduces the disparity between demographic groups. Thus, we introduce a quantity that summarizes the fairness-utility trade-off, called the causal fairness/utility ratio, defined as the ratio of the reduction in discrimination vs. the excess loss from constraining a causal pathway. This quantity is suitable for comparing the fairness-utility trade-off across causal pathways. Finally, as our approach requires causally-constrained fair predictors, we introduce a new neural approach for causally-constrained fair learning.

Motivation & Objective

• To analyze the trade-off between fairness and accuracy in machine learning from a causal perspective, addressing limitations of traditional statistical fairness definitions.
• To quantify the accuracy cost of enforcing fairness along specific causal pathways using a novel metric called path-specific excess loss (PSEL).
• To decompose the total excess loss from fairness constraints into contributions from individual causal pathways, enabling granular analysis of fairness-utility trade-offs.
• To introduce the causal fairness/utility ratio as a comparative metric for evaluating the efficiency of fairness interventions across different causal pathways.
• To develop a neural architecture for learning causally constrained fair predictors, enabling practical deployment of the proposed framework.

Proposed method

• Proposes path-specific excess loss (PSEL) as a measure of how much a predictor's loss increases when a specific causal pathway is constrained to be fair.
• Derives a decomposition of the total excess loss (TEL) into a sum of PSELs across all causal pathways, enabling attribution of accuracy costs to individual paths.
• Introduces the causal fairness/utility ratio as the ratio of discrimination reduction to excess loss, allowing comparison of fairness interventions across pathways.
• Develops a neural network-based method for training causally constrained fair predictors by modifying backpropagation to enforce fairness along specified causal paths.
• Applies an algorithm (Alg. 1) that systematically evaluates the impact of removing each causal pathway on fairness (measured by TVD) and accuracy (measured by loss), using PSEL and CFUR metrics.
• Uses causal graphs to visualize the contribution of each pathway to fairness and accuracy, with PSEL and TVD values annotated on the graph.

Figure 1 : TV vs. Excess Loss.

Experimental results

Research questions

• RQ1 How can the fairness-accuracy trade-off be quantified along individual causal pathways rather than globally?
• RQ2 To what extent does enforcing fairness along a specific causal pathway increase the predictor's loss, and how can this cost be measured?
• RQ3 Which causal pathways offer the most favorable fairness-utility trade-offs, and how can they be compared?
• RQ4 Can a neural network be trained to enforce fairness along specific causal pathways while maintaining predictive performance?
• RQ5 How do different causal pathways contribute to group disparities in real-world machine learning applications?

Key findings

• In the COMPAS recidivism dataset, removing the spurious effect reduced group disparity by 3% (TVD), while removing the indirect effect reduced it by 5%, indicating stronger fairness gains from indirect effect removal.
• The path-specific excess loss (PSEL) for the indirect effect in COMPAS was 12% of AUROC, significantly higher than the 6% for the spurious effect, indicating a larger accuracy cost for indirect effect constraints.
• The causal fairness/utility ratio (CFUR) analysis showed that removing the direct effect yielded the best trade-off between fairness gain and accuracy loss in the German credit dataset.
• The total excess loss (TEL) was successfully decomposed into a sum of PSELs across pathways, validating the theoretical decomposition (Theorem 1).
• The proposed neural method for causally constrained fair learning enabled the construction of fair predictors that could be evaluated using the PSEL and CFUR framework across multiple datasets.
• Empirical results across three real-world datasets (German credit, COMPAS, random forest) consistently showed that fairness and accuracy are in tension, with no pathway offering perfect fairness without some accuracy cost.

(a) (A)PSEL and (A)TVD values.