[Paper Review] Causal foundations of bias, disparity and fairness

V. A. Traag, L. Waltman|arXiv (Cornell University)|Jul 27, 2022

Social and Intergroup Psychology18 citations

TL;DR

This paper proposes a causal framework defining bias as an unjustified direct causal effect and disparity as a direct or indirect causal effect that includes such a bias. Using structural causal models, it clarifies the distinction between bias and disparity, demonstrates their role in real-world cases like gender bias in science and racial bias in policing, and argues that fairness requires ethical judgment beyond data alone.

ABSTRACT

The study of biases, such as gender or racial biases, is an important topic in the social and behavioural sciences. However, the literature does not always clearly define the concept. Definitions of bias are often ambiguous or not provided at all. To study biases in a precise manner, it is important to have a well-defined concept of bias. We propose to define bias as a direct causal effect that is unjustified. We propose to define the closely related concept of disparity as a direct or indirect causal effect that includes a bias. Our proposed definitions can be used to study biases and disparities in a more rigorous and systematic way. We compare our definitions of bias and disparity with various criteria of fairness introduced in the artificial intelligence literature. In addition, we discuss how our definitions relate to discrimination. We illustrate our definitions of bias and disparity in two case studies, focusing on gender bias in science and racial bias in police shootings. Our proposed definitions aim to contribute to a better appreciation of the causal intricacies of studies of biases and disparities. We hope that this will also promote an improved understanding of the policy implications of such studies.

Motivation & Objective

• To address the ambiguity and imprecision in existing definitions of bias and disparity in social and behavioral sciences.
• To establish a rigorous, causal foundation for understanding bias and disparity using structural causal models.
• To clarify the ethical dimension of fairness by distinguishing empirical causality from normative judgments about justification.
• To demonstrate how feedback loops in predictive systems can generate spurious disparities that appear self-validating.
• To guide policy interventions by identifying where in a causal pathway a bias originates, enabling more effective and targeted corrections.

Proposed method

• Defines bias as a direct causal effect that is ethically unjustified, using structural causal models (SCMs) to formalize causal relationships.
• Defines disparity as any direct or indirect causal effect from a variable X to Y that includes at least one bias, making Y unfair with respect to X.
• Applies counterfactual reasoning to assess fairness, distinguishing between causal effects and spurious correlations.
• Uses case studies—gender bias in academic hiring and racial bias in police shootings—to illustrate how biases and disparities emerge in real systems.
• Compares proposed definitions with fairness criteria from AI literature, highlighting incompatibilities with data-only fairness metrics.
• Emphasizes that acting on predictions introduces feedback loops, transforming prediction into causation and reinforcing disparities.

Experimental results

Research questions

• RQ1 How can bias and disparity be formally defined in a way that distinguishes causal effects from normative judgments?
• RQ2 In what ways do feedback loops in predictive systems reproduce or amplify disparities, even when the original prediction is based on biased data?
• RQ3 Why are common AI fairness criteria incompatible with the proposed causal definitions of fairness?
• RQ4 How can policy interventions be more effective when they target the root cause of bias in a causal pathway rather than downstream outcomes?
• RQ5 To what extent can causal modeling help identify whether observed differences are due to bias, disparity, or neither?

Key findings

• Bias is formally defined as a direct causal effect that is considered unjustified, requiring ethical evaluation beyond statistical analysis.
• Disparity is defined as any causal pathway from X to Y that includes at least one bias, making the outcome unfair with respect to X.
• In the case of police shootings, a prediction model that uses stop rates as input can create a self-reinforcing loop where racial disparities in stops lead to racially skewed predictions, which then justify further stops.
• The paper shows that even 'just for prediction' AI systems can induce causal effects when used in decision-making, leading to feedback loops that entrench disparities.
• Common AI fairness criteria such as demographic parity or equal opportunity are incompatible with the proposed definitions, as they may classify biased systems as fair if they produce balanced outcomes.
• Without a proper causal understanding, policy interventions such as affirmative action may address symptoms rather than root causes, potentially leading to unintended consequences.