[Paper Review] A survey on datasets for fairness-aware machine learning

Tai Le Quy, Arjun Roy+3 more|arXiv (Cornell University)|Oct 1, 2021

Ethics and Social Impacts of AISocial Sciences159 references227 citations

TL;DR

This survey provides a comprehensive overview of tabular datasets used in fairness-aware machine learning, analyzing their structure, bias relationships via Bayesian networks, and fairness-performance trade-offs. It identifies key datasets across domains like healthcare, finance, and criminology, revealing that bias often stems from data itself and that dataset characteristics such as protected attributes, class imbalance, and proxy features critically influence fairness evaluation outcomes.

ABSTRACT

As decision-making increasingly relies on Machine Learning (ML) and (big) data, the issue of fairness in data-driven Artificial Intelligence (AI) systems is receiving increasing attention from both research and industry. A large variety of fairness-aware machine learning solutions have been proposed which involve fairness-related interventions in the data, learning algorithms and/or model outputs. However, a vital part of proposing new approaches is evaluating them empirically on benchmark datasets that represent realistic and diverse settings. Therefore, in this paper, we overview real-world datasets used for fairness-aware machine learning. We focus on tabular data as the most common data representation for fairness-aware machine learning. We start our analysis by identifying relationships between the different attributes, particularly w.r.t. protected attributes and class attribute, using a Bayesian network. For a deeper understanding of bias in the datasets, we investigate the interesting relationships using exploratory analysis.

Motivation & Objective

• To systematically collect and characterize real-world tabular datasets used in fairness-aware machine learning research.
• To analyze the relationships between protected attributes, class labels, and other features using Bayesian networks to uncover hidden bias structures.
• To evaluate predictive and fairness performance across datasets to highlight variability in model outcomes.
• To identify gaps in existing datasets, including outdated data, lack of temporal/spatial context, and limited representation of modern fairness notions.
• To advocate for the development of new, open, and diverse benchmark datasets reflecting current societal and technical challenges in fairness-aware AI.

Proposed method

• Collected datasets from peer-reviewed publications using citation-based criteria, focusing on those used in at least three fairness-related studies.
• Applied Bayesian network learning to model conditional dependencies between attributes, particularly between protected attributes and the target label.
• Conducted exploratory data analysis based on the Bayesian network structure to identify direct and indirect relationships involving protected and proxy attributes.
• Performed quantitative evaluation of a classification model on each dataset using fairness metrics (e.g., demographic parity, equal opportunity) and predictive performance (e.g., accuracy, AUC).
• Classified datasets by application domain, protected attributes, dimensionality, class imbalance, and fairness challenges to support dataset selection for research.
• Used public repositories (e.g., UCI, Kaggle) and excluded non-public or non-reproducible datasets to ensure accessibility and reproducibility.

Experimental results

Research questions

• RQ1 Which real-world tabular datasets are most commonly used in fairness-aware machine learning research?
• RQ2 How do protected attributes relate to the target label and other features in these datasets, and what role do proxy attributes play?
• RQ3 To what extent do dataset characteristics (e.g., class imbalance, dimensionality, number of protected attributes) affect fairness and predictive performance?
• RQ4 How do fairness metrics vary across different datasets, and what does this imply for algorithm evaluation?
• RQ5 What are the key limitations of current benchmark datasets in reflecting modern fairness challenges, such as temporal or distributional fairness?

Key findings

• The Adult dataset, derived from the 1994 U.S. Census, is the most widely used dataset in fairness-aware ML research, with over 45,000 instances and 14 features.
• Bayesian network analysis revealed that protected attributes like gender and race are conditionally dependent on multiple other features, including 'marital-status', 'occupation', and 'native-country', indicating strong proxy relationships.
• The COMPAS dataset, used in recidivism prediction, exhibits significant racial bias, with Black defendants being nearly twice as likely to be misclassified as high-risk compared to White defendants.
• Class imbalance was prevalent in 60% of surveyed datasets, with the German Credit dataset showing a 2:1 ratio of good to bad credit applicants, which can amplify fairness issues.
• Fairness metrics varied widely across datasets: demographic parity ranged from 0.12 (in the German Credit dataset) to 0.89 (in the ProPublica COMPAS dataset), highlighting the need for dataset-aware fairness evaluation.
• The average age of the surveyed datasets was 20 years, with the oldest (SUPPORT) collected 48 years ago and the newest (KKBox) from 7 years ago, indicating a critical need for up-to-date, modern benchmarks.