[Paper Review] On Fairness and Calibration

Geoff Pleiss, Manish Raghavan+3 more|arXiv (Cornell University)|Sep 6, 2017

Explainable Artificial Intelligence (XAI)Computer Science288 citations

TL;DR

The paper analyzes the tension between minimizing error disparity across population groups and maintaining calibrated probability estimates, proving calibration is incompatible with broad equalized-odds constraints and proposing a calibration-preserving relaxation with an information-withholding post-processing method to achieve equal-cost constraints.

ABSTRACT

The machine learning community has become increasingly concerned with the potential for bias and discrimination in predictive models. This has motivated a growing line of work on what it means for a classification procedure to be "fair." In this paper, we investigate the tension between minimizing error disparity across different population groups while maintaining calibrated probability estimates. We show that calibration is compatible only with a single error constraint (i.e. equal false-negatives rates across groups), and show that any algorithm that satisfies this relaxation is no better than randomizing a percentage of predictions for an existing classifier. These unsettling findings, which extend and generalize existing results, are empirically confirmed on several datasets.

Motivation & Objective

• Motivate the study of fairness in predictive classifications in sensitive domains.
• Formalize calibration and Equalized Odds for probabilistic classifiers across groups with different base rates.
• Investigate whether calibration can coexist with relaxed error-rate constraints and characterize feasibility.
• Provide a post-processing method to achieve calibration with an equal-cost constraint and analyze its implications.
• Empirically evaluate calibrated fairness versus uncalibrated approaches on real datasets.

Proposed method

• Extend the Equalized Odds framework to probabilistic classifiers with two groups and different base rates.
• Define generalized false-positive and false-negative rates for calibrated probabilistic outputs.
• Characterize calibration constraints as linear relations between FP and FN rates (lines in FP/FN space).
• Show impossibility results: calibration plus Equalized Odds generally only possible when classifiers are perfect.
• Introduce a relaxed Equalized Odds with Calibration using a cost function g_t that combines FP and FN with group-specific weights.
• Propose a post-processing algorithm that randomizes a portion of predictions to match equalized costs while preserving calibration.
• Prove feasibility conditions and analyze optimality of the information-withholding approach.
• Provide theoretical results including an impossibility theorem for multiple equal-cost constraints and a unique feasible solution under the relaxed framework.

Experimental results

Research questions

• RQ1 Can calibrated probabilistic classifiers satisfy Equalized Odds across groups with different base rates?
• RQ2 What relaxations of Equalized Odds can preserve calibration, and under what conditions are they feasible?
• RQ3 Is there a post-processing method that achieves calibration with an equal-cost constraint, and what are its implications for fairness and individual utility?
• RQ4 What happens empirically when comparing calibrated fairness against uncalibrated Equalized Odds on real-world datasets?

Key findings

• Calibration and Equalized Odds are generally incompatible except in trivial (perfect) cases.
• A relaxed framework with a single equal-cost constraint can be achieved while maintaining calibration, via post-processing that withholds certain predictions.
• There exists a unique solution under the relaxed constraint, achieved by randomizing predictions to equalize cost across groups.
• A general impossibility shows that satisfying multiple distinct equal-cost constraints with calibration forces perfect accuracy.
• Empirical evaluations across income prediction, health prediction, and recidivism datasets illustrate trade-offs: calibration can increase overall error and shifting disparities depending on the scenario.