[Paper Review] Software Fairness: An Analysis and Survey

Ezekiel Soremekun, Mike Papadakis+2 more|arXiv (Cornell University)|May 18, 2022

Ethics and Social Impacts of AI9 citations

TL;DR

This paper conducts a comprehensive analysis of 164 studies on fairness in learning-based software systems, categorizing approaches by fairness measure, task type, analysis method, and access level (black/white/grey box). It reveals critical gaps—especially in fairness specification, intersectional/sequential fairness, unstructured data (e.g., text, images), and white-box in-processing techniques—highlighting underexplored challenges in policy-based and human-in-the-loop bias mitigation.

ABSTRACT

In the last decade, researchers have studied fairness as a software property. In particular, how to engineer fair software systems? This includes specifying, designing, and validating fairness properties. However, the landscape of works addressing bias as a software engineering concern is unclear, i.e., techniques and studies that analyze the fairness properties of learning-based software. In this work, we provide a clear view of the state-of-the-art in software fairness analysis. To this end, we collect, categorize and conduct an in-depth analysis of 164 publications investigating the fairness of learning-based software systems. Specifically, we study the evaluated fairness measure, the studied tasks, the type of fairness analysis, the main idea of the proposed approaches, and the access level (e.g., black, white, or grey box). Our findings include the following: (1) Fairness concerns (such as fairness specification and requirements engineering) are under-studied; (2) Fairness measures such as conditional, sequential, and intersectional fairness are under-explored; (3) Unstructured datasets (e.g., audio, image, and text) are barely studied for fairness analysis; and (4) Software fairness analysis techniques hardly employ white-box, in-processing machine learning (ML) analysis methods. In summary, we observed several open challenges including the need to study intersectional/sequential bias, policy-based bias handling, and human-in-the-loop, socio-technical bias mitigation.

Motivation & Objective

• To clarify the current landscape of fairness in software engineering, particularly in learning-based systems.
• To identify under-researched areas in fairness specification, requirements engineering, and bias mitigation.
• To analyze the distribution of fairness measures, tasks, and analysis techniques across existing literature.
• To evaluate the prevalence and effectiveness of different access levels (black, white, grey box) in fairness analysis.
• To highlight open challenges in intersectional, sequential, and socio-technical bias handling in software fairness.

Proposed method

• Systematic collection and categorization of 164 publications on fairness in learning-based software systems.
• Classification of studies based on fairness measure (e.g., demographic parity, equal opportunity), task type (e.g., classification, NLP), and analysis approach.
• Categorization of analysis techniques by access level: black-box (input-output), grey-box (partial model access), and white-box (full model inspection).
• In-depth analysis of the main ideas behind proposed fairness techniques, including preprocessing, in-processing, and post-processing methods.
• Identification of trends and gaps through thematic and quantitative analysis of the collected literature.
• Focus on under-explored areas such as sequential and intersectional fairness, unstructured data (audio, image, text), and in-processing ML techniques.

Experimental results

Research questions

• RQ1 What fairness measures are most commonly used in software fairness research, and which are under-explored?
• RQ2 Which software tasks (e.g., NLP, computer vision) are most frequently studied in fairness analysis, and which are under-represented?
• RQ3 How prevalent are white-box, grey-box, and black-box analysis techniques in fairness evaluation of learning-based systems?
• RQ4 To what extent are fairness specification and requirements engineering addressed in existing software fairness literature?
• RQ5 What are the key open challenges in mitigating intersectional, sequential, and policy-based bias in software fairness?

Key findings

• Fairness concerns such as fairness specification and requirements engineering are significantly under-studied in the literature.
• Fairness measures like conditional, sequential, and intersectional fairness are under-explored despite their importance in real-world fairness applications.
• Unstructured data types—such as audio, images, and text—are rarely studied in fairness analysis, despite their prevalence in ML systems.
• White-box in-processing machine learning techniques for fairness analysis are scarcely employed, indicating a gap in model-level intervention strategies.
• There is a notable lack of research on policy-based bias handling and human-in-the-loop approaches for socio-technical fairness mitigation.
• The majority of fairness analysis remains black-box, limiting deep understanding of model behavior and bias propagation mechanisms.