[Paper Review] Identifying Biased Subgroups in Ranking and Classification

Eliana Pastor, Luca de Alfaro+1 more|arXiv (Cornell University)|Aug 17, 2021

Imbalanced Data Classification Techniques21 references4 citations

TL;DR

This paper proposes a method to automatically detect biased data subgroups in classification and ranking systems by measuring statistical divergence from global model performance. Using frequent itemset mining and Shapley values, it identifies subgroups with extreme FPR/FNR or benefit disparities and quantifies the contribution of each attribute to the bias, enabling targeted fairness audits without prior knowledge of sensitive attributes.

ABSTRACT

When analyzing the behavior of machine learning algorithms, it is important to identify specific data subgroups for which the considered algorithm shows different performance with respect to the entire dataset. The intervention of domain experts is normally required to identify relevant attributes that define these subgroups. We introduce the notion of divergence to measure this performance difference and we exploit it in the context of (i) classification models and (ii) ranking applications to automatically detect data subgroups showing a significant deviation in their behavior. Furthermore, we quantify the contribution of all attributes in the data subgroup to the divergent behavior by means of Shapley values, thus allowing the identification of the most impacting attributes.

Motivation & Objective

• To address the challenge of identifying data subgroups where machine learning models exhibit significantly different performance compared to the global dataset.
• To automate the detection of biased subgroups without requiring prior knowledge of sensitive attributes or predefined protected groups.
• To quantify the contribution of individual attributes to subgroup divergence using Shapley values for actionable insights.
• To extend the approach to both classification and ranking scenarios, enabling fairness auditing in diverse ML applications.

Proposed method

• Introduces a divergence metric to measure the difference in performance statistics (e.g., FPR, FNR, benefit) between a subgroup and the global dataset.
• Employs frequent itemset mining (e.g., FP-growth) to efficiently discover subgroups defined by combinations of attribute values above a support threshold.
• Computes outcome statistics (e.g., average benefit, false positive rate) for each itemset during mining to determine divergence.
• Applies Shapley values to decompose the total divergence of an itemset into contributions from each individual attribute value.
• Uses a support threshold to filter subgroups, ensuring they are large enough to be statistically meaningful and analyzable.
• Extends the framework to ranking systems by modeling benefit as a function of rank position (e.g., γ(i) = i^−0.1), enabling divergence analysis of rank-based outcomes.

Experimental results

Research questions

• RQ1 Which data subgroups exhibit significantly different performance (e.g., higher FPR or FNR) compared to the global model behavior?
• RQ2 How can we automatically detect such subgroups without relying on predefined sensitive attributes?
• RQ3 Which individual attributes contribute most to the observed divergence in a given subgroup?
• RQ4 How does the model’s performance vary across subgroups in ranking scenarios, particularly in terms of benefit distribution?

Key findings

• In the COMPAS dataset, the subgroup {age<25, #prior>3, sex=Male} showed a 59% divergence in FPR, indicating a high rate of false positives.
• The subgroup {age>45, charge=M, race=Cauc} exhibited a 30% divergence in FNR, suggesting a high rate of false negatives for this demographic.
• Shapley value analysis revealed that age<25 and #prior>3 contributed most to the FPR divergence in the {age<25, #prior>3, sex=Male} subgroup, while sex=Male had a minor role.
• In the Law School Dataset, the itemset {LSAT≤33.0, race=Black, sex=Male} showed the lowest ZFYA (first-year average grade) divergence, with race=Black being the dominant factor in the negative outcome.
• The method efficiently identified all relevant itemsets above the support threshold (s=0.005) and computed their divergences in under a dozen seconds, even with 10^5 itemsets.
• The analysis highlighted race as a predominant factor in unfair benefit distribution in the Law School internship ranking, signaling a critical area for further investigation.