[Paper Review] Fairness Aware Counterfactuals for Subgroups

Loukas Kavouras, Konstantinos Tsopelas+7 more|arXiv (Cornell University)|Jun 26, 2023

Social and Intergroup PsychologySocial Sciences3 citations

TL;DR

FACTS introduces a model-agnostic framework for auditing subgroup fairness by analyzing counterfactual recourse, offering refined notions of fairness beyond mean cost, including effectiveness-cost trade-offs and budget-constrained recourse. It enables robust, parameterized evaluation of fairness across subgroups using cumulative cost distributions and conditional metrics, demonstrating superior detection of hidden disparities in benchmark datasets.

ABSTRACT

In this work, we present Fairness Aware Counterfactuals for Subgroups (FACTS), a framework for auditing subgroup fairness through counterfactual explanations. We start with revisiting (and generalizing) existing notions and introducing new, more refined notions of subgroup fairness. We aim to (a) formulate different aspects of the difficulty of individuals in certain subgroups to achieve recourse, i.e. receive the desired outcome, either at the micro level, considering members of the subgroup individually, or at the macro level, considering the subgroup as a whole, and (b) introduce notions of subgroup fairness that are robust, if not totally oblivious, to the cost of achieving recourse. We accompany these notions with an efficient, model-agnostic, highly parameterizable, and explainable framework for evaluating subgroup fairness. We demonstrate the advantages, the wide applicability, and the efficiency of our approach through a thorough experimental evaluation of different benchmark datasets.

Motivation & Objective

• To address the limitations of mean-cost-based fairness metrics in assessing recourse burden across protected subgroups.
• To introduce nuanced, robust fairness notions that capture micro-level individual variation and macro-level subgroup behavior in recourse costs.
• To develop a model-agnostic, parameterizable framework for evaluating subgroup fairness using counterfactual explanations.
• To enable fairness auditing that accounts for realistic constraints, such as budget limits on recourse actions.
• To demonstrate the framework’s efficiency, wide applicability, and ability to detect hidden disparities in real-world datasets.

Proposed method

• Introduces the effectiveness-cost distribution (ECD) to visualize and analyze the trade-off between recourse cost and success rate across individuals.
• Proposes new fairness metrics: Equal Cost of Effectiveness (Macro/Micro), Equal Choice for Recourse, Equal Effectiveness within Budget, and Fair Effectiveness-Cost Trade-Off.
• Uses counterfactual explanations to compute the minimal feature perturbations required for an individual to achieve a favorable outcome.
• Employs a parameterized evaluation pipeline that allows tuning thresholds for cost, effectiveness, and budget constraints.
• Applies fp-growth to identify subgroups in the affected population and ranks them based on fairness metric scores.
• Integrates conditional mean recourse and fairness-aware ranking to detect subgroups with disproportionate burden.

Experimental results

Research questions

• RQ1 How do existing fairness metrics based on mean recourse cost fail to capture the true burden distribution across subgroups?
• RQ2 What refined fairness notions can better capture the equity of recourse access at both individual and subgroup levels?
• RQ3 How can fairness auditing be made robust to cost constraints and realistic actionability thresholds?
• RQ4 To what extent can the proposed framework detect hidden disparities not revealed by standard fairness metrics?
• RQ5 How does the framework perform across diverse real-world datasets in identifying unfairly burdened subgroups?

Key findings

• In the Ad Campaign dataset, 128 out of the top 10% most unfair subgroups showed bias against females under the Fair Effectiveness-Cost Trade-Off metric, highlighting the method’s sensitivity to hidden disparities.
• The Equal Effectiveness within Budget (1.0) metric identified only 1 subgroup as highly unfair, indicating that most subgroups could achieve recourse under low-cost constraints.
• Under the Equal Choice for Recourse metric with threshold 0.3, 66 subgroups showed bias against females, demonstrating that even small-cost actions can reveal significant inequity.
• The Equal Cost of Effectiveness (Macro) metric with threshold 0.7 detected 264 of the most unfair subgroups, showing strong sensitivity to moderate-cost recourse.
• The Fair Effectiveness-Cost Trade-Off metric achieved the highest average ranking score (0.971) across all fairness metrics, indicating superior overall fairness detection capability.
• In the credit assessment dataset, the framework detected that 80% of race 1 individuals could achieve recourse under a cost budget of 2, compared to only 60% of race 0, revealing a previously undetected inequity.