[Paper Review] Information Theoretic Measures for Fairness-aware Feature Selection

Sajad Khodadadian, Mohamed Nafea+2 more|arXiv (Cornell University)|Jun 1, 2021

Ethics and Social Impacts of AI40 references5 citations

TL;DR

This paper proposes an information-theoretic framework for fairness-aware feature selection that quantifies each feature’s marginal impact on both prediction accuracy and discrimination using Shapley values. It introduces joint accuracy and discrimination coefficients derived from mutual information, enabling selection of features that balance utility and fairness without relying on a specific classifier, validated on synthetic and COMPAS datasets with measurable reductions in bias upon removing high-impact features like Age and Prior Counts.

ABSTRACT

Machine learning algorithms are increasingly used for consequential decision making regarding individuals based on their relevant features. Features that are relevant for accurate decisions may however lead to either explicit or implicit forms of discrimination against unprivileged groups, such as those of certain race or gender. This happens due to existing biases in the training data, which are often replicated or even exacerbated by the learning algorithm. Identifying and measuring these biases at the data level is a challenging problem due to the interdependence among the features, and the decision outcome. In this work, we develop a framework for fairness-aware feature selection which takes into account the correlation among the features and the decision outcome, and is based on information theoretic measures for the accuracy and discriminatory impacts of features. In particular, we first propose information theoretic measures which quantify the impact of different subsets of features on the accuracy and discrimination of the decision outcomes. We then deduce the marginal impact of each feature using Shapley value function; a solution concept in cooperative game theory used to estimate marginal contributions of players in a coalitional game. Finally, we design a fairness utility score for each feature (for feature selection) which quantifies how this feature influences accurate as well as nondiscriminatory decisions. Our framework depends on the joint statistics of the data rather than a particular classifier design. We examine our proposed framework on real and synthetic data to evaluate its performance.

Motivation & Objective

• Address the challenge of identifying features that contribute to both accurate predictions and discriminatory outcomes in machine learning systems.
• Develop a method to quantify the marginal impact of features on accuracy and discrimination while accounting for feature interdependencies.
• Create a fairness-utility score that enables feature selection under trade-offs between predictive performance and fairness.
• Ensure the framework is classifier-agnostic by relying only on joint statistical distributions of features, sensitive attributes, and outcomes.
• Evaluate the framework on real-world (COMPAS) and synthetic datasets to demonstrate its effectiveness in reducing bias without significant accuracy loss.

Proposed method

• Define two information-theoretic measures: one for marginal accuracy impact (φ^Acc) and one for marginal discrimination impact (φ^D), based on mutual information between feature subsets, sensitive attributes, and outcomes.
• Use the Shapley value from cooperative game theory to compute the marginal contribution of each individual feature to the overall accuracy and discrimination scores.
• Construct a fairness-utility score as a weighted combination of φ^Acc and φ^D, with hyperparameter α balancing accuracy and fairness objectives.
• Compute the joint probability distribution from data (or a graphical model in synthetic cases) to derive all required mutual information terms.
• Apply the framework to real and synthetic datasets by training a neural network classifier and measuring changes in prediction error and bias (KL divergence) when features are removed.
• Use confidence intervals from 100 repeated experiments to assess stability of the measures.

Experimental results

Research questions

• RQ1 How can we quantify the marginal contribution of each feature to both prediction accuracy and discriminatory impact in a way that accounts for feature interdependencies?
• RQ2 To what extent can information-theoretic measures capture non-linear relationships between features, sensitive attributes, and decision outcomes?
• RQ3 Can Shapley value-based decomposition effectively isolate the individual impact of features on fairness and utility in a cooperative framework?
• RQ4 How do the proposed fairness-utility scores compare to existing methods in identifying high-impact discriminatory proxies in real-world datasets?
• RQ5 Does removing features with high discrimination coefficients lead to measurable reductions in classifier bias without significant loss in accuracy?

Key findings

• In the synthetic dataset, features X₃ and X₄ had the highest marginal accuracy coefficients (4.467 and 3.122, respectively), and their removal led to the largest increases in prediction error.
• Feature X₁ had the highest marginal discrimination coefficient (2.7927 × 10⁷), and its removal resulted in the lowest bias in the classifier output.
• On the COMPAS dataset, Age and Prior Counts had the highest discrimination coefficients (6.0904 × 10⁶ and 5.0466 × 10⁶), respectively, and their removal most significantly reduced bias.
• Charge Degree and Gender had the lowest accuracy coefficients (0.1723 and 0.4186), and their removal had minimal impact on prediction error, confirming their low utility for classification.
• The framework successfully identified Age and Prior Counts as strong proxies for race in the COMPAS dataset, aligning with prior findings on redlining and proxy discrimination.
• The results from both datasets show strong correlation between the proposed coefficients and actual changes in classifier performance and bias, validating the framework’s reliability and interpretability.