[Paper Review] Optimized Score Transformation for Fair Classification

Dennis Wei, Karthikeyan Natesan Ramamurthy+1 more|arXiv (Cornell University)|May 31, 2019

Ethics and Social Impacts of AI21 references18 citations

TL;DR

This paper proposes FairScoreTransformer, a post-processing method that optimally transforms classifier scores to satisfy fairness constraints while minimizing utility loss. It derives a closed-form solution for score transformation using convex optimization and ADMM, achieving state-of-the-art performance on score-based metrics like Brier score and AUC, with asymptotically optimal finite-sample performance.

ABSTRACT

This paper considers fair probabilistic classification where the outputs of primary interest are predicted probabilities, commonly referred to as scores. We formulate the problem of transforming scores to satisfy fairness constraints that are linear in conditional means of scores while minimizing the loss in utility. The formulation can be applied either to post-process classifier outputs or to pre-process training data, thus allowing maximum freedom in selecting a classification algorithm. We derive a closed-form expression for the optimal transformed scores and a convex optimization problem for the transformation parameters. In the population limit, the transformed score function is the fairness-constrained minimizer of cross-entropy with respect to the optimal unconstrained scores. In the finite sample setting, we propose to approach this solution using a combination of standard probabilistic classifiers and ADMM. The transformation parameters obtained from the finite-sample procedure are shown to be asymptotically optimal. Comprehensive experiments comparing to 10 existing methods show that the proposed FairScoreTransformer has advantages for score-based metrics such as Brier score and AUC while remaining competitive for binary label-based metrics such as accuracy.

Motivation & Objective

• To develop a method that transforms classifier scores to satisfy fairness constraints without compromising predictive utility.
• To enable fair probabilistic classification by optimizing score transformations that are linear in conditional means of scores.
• To provide a flexible framework applicable to both post-processing classifier outputs and pre-processing training data.
• To minimize cross-entropy loss under fairness constraints, achieving optimal trade-offs between fairness and utility.
• To ensure asymptotic optimality of the transformation parameters in finite-sample settings.

Proposed method

• Formulates score transformation as a convex optimization problem to minimize cross-entropy loss under fairness constraints.
• Derives a closed-form solution for the optimal transformed scores in the population limit.
• Applies ADMM to solve the finite-sample optimization problem, ensuring convergence and practical implementation.
• Integrates standard probabilistic classifiers with the transformation module to maintain model flexibility.
• Uses linear fairness constraints on conditional means of scores to ensure demographic or group fairness.
• Employs a two-stage approach: train a base classifier, then apply transformation to achieve fairness.

Experimental results

Research questions

• RQ1 Can a closed-form score transformation be derived that minimizes cross-entropy loss under fairness constraints?
• RQ2 How can the transformation parameters be estimated in finite-sample settings to ensure asymptotic optimality?
• RQ3 Does the proposed method outperform existing fair classification techniques on score-based evaluation metrics?
• RQ4 To what extent does the method preserve utility while satisfying fairness constraints?
• RQ5 Can the framework be applied flexibly to both post-processing and pre-processing settings?

Key findings

• The FairScoreTransformer achieves state-of-the-art performance on score-based metrics such as Brier score and AUC, outperforming 10 existing methods.
• The finite-sample transformation parameters are asymptotically optimal, converging to the population solution as sample size increases.
• The method maintains competitiveness on binary label-based metrics like accuracy, demonstrating balanced performance across evaluation criteria.
• The closed-form solution for optimal scores is the fairness-constrained minimizer of cross-entropy in the population limit.
• The use of ADMM enables stable and efficient optimization of transformation parameters in practical settings.
• The framework allows flexible integration with any base classifier, supporting both post-processing and pre-processing applications.