[Paper Review] Inherent Trade-Offs in the Fair Determination of Risk Scores

Jon Kleinberg, Sendhil Mullainathan+1 more|arXiv (Cornell University)|Sep 19, 2016

Bayesian Modeling and Causal InferenceComputer Science422 citations

TL;DR

The paper proves that three natural fairness conditions for probabilistic risk scores cannot be satisfied simultaneously except in special cases, revealing inherent trade-offs in fair risk assessment.

ABSTRACT

Recent discussion in the public sphere about algorithmic classification has involved tension between competing notions of what it means for a probabilistic classification to be fair to different groups. We formalize three fairness conditions that lie at the heart of these debates, and we prove that except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously. Moreover, even satisfying all three conditions approximately requires that the data lie in an approximate version of one of the constrained special cases identified by our theorem. These results suggest some of the ways in which key notions of fairness are incompatible with each other, and hence provide a framework for thinking about the trade-offs between them.

Motivation & Objective

• Formalize three widely discussed fairness conditions for probabilistic classifications.
• Show that these conditions cannot be satisfied simultaneously in general.
• Characterize the precise scenarios where all three can hold together.
• Provide a framework to reason about trade-offs in fairness definitions for risk scores.

Proposed method

• Define a two-group model with feature vectors and group-specific distributions.
• Introduce risk assignments via bins with scores and a mapping from features to bins.
• Formalize calibration within groups and balance for positive/negative classes as fairness conditions.
• Prove a characterization theorem showing when all three conditions can hold together.
• Extend to approximate versions of fairness and discuss implications.

Experimental results

Research questions

• RQ1 Can calibration within groups, balance for the positive class, and balance for the negative class be satisfied simultaneously?
• RQ2 Under what data-generating scenarios do these fairness conditions coincide with perfect prediction or equal base rates?
• RQ3 How do approximate versions of the fairness conditions constrain achievable risk score constructions?
• RQ4 What do these trade-offs imply for designing fair risk scoring systems in practice?

Key findings

• All three fairness conditions are incompatible in general unless the data exhibit perfect prediction or equal base rates.
• Approximately satisfying the conditions forces the risk scores to resemble one of the two constrained special cases.
• The results hold independently of the specific method used to compute the risk scores (algorithmic or human).
• A continuous function governs the relationship between approximation level and proximity to the special cases.
• The work provides a theoretical framework for understanding trade-offs among common fairness notions.