[Paper Review] Directional Bias Amplification

Angelina Wang, Olga Russakovsky|arXiv (Cornell University)|Feb 24, 2021

Adversarial Robustness in Machine Learning61 references20 citations

TL;DR

This paper introduces Directional Bias Amplification (BiasAmp→), a new metric that disentangles bias amplification into distinct directional flows—A→T and T→A—addressing limitations of prior metrics that conflated these paths. The method improves fairness evaluation by accounting for base rates and enabling targeted interventions, with empirical results showing that models can exhibit low bias amplification yet still have high fairness disparities, especially in risk prediction settings.

ABSTRACT

Mitigating bias in machine learning systems requires refining our understanding of bias propagation pathways: from societal structures to large-scale data to trained models to impact on society. In this work, we focus on one aspect of the problem, namely bias amplification: the tendency of models to amplify the biases present in the data they are trained on. A metric for measuring bias amplification was introduced in the seminal work by Zhao et al. (2017); however, as we demonstrate, this metric suffers from a number of shortcomings including conflating different types of bias amplification and failing to account for varying base rates of protected attributes. We introduce and analyze a new, decoupled metric for measuring bias amplification, $ ext{BiasAmp}_{ ightarrow}$ (Directional Bias Amplification). We thoroughly analyze and discuss both the technical assumptions and normative implications of this metric. We provide suggestions about its measurement by cautioning against predicting sensitive attributes, encouraging the use of confidence intervals due to fluctuations in the fairness of models across runs, and discussing the limitations of what this metric captures. Throughout this paper, we work to provide an interrogative look at the technical measurement of bias amplification, guided by our normative ideas of what we want it to encompass. Code is located at https://github.com/princetonvisualai/directional-bias-amp

Motivation & Objective

• To address the shortcomings of existing bias amplification metrics, which conflate different types of bias and fail to account for base rates of protected attributes.
• To develop a more precise and normatively informed metric that distinguishes between A→T and T→A bias amplification pathways.
• To guide fairer model evaluation by clarifying which types of bias amplification are actionable and contextually relevant.
• To caution against overreliance on bias amplification as a sole fairness metric, especially in settings where training data correlations may reflect systemic inequities.
• To encourage the use of confidence intervals and careful interpretation of fairness metrics due to model variability across random seeds.

Proposed method

• Proposes a new metric, BiasAmp→, which quantifies bias amplification in a directional manner, distinguishing between amplification from protected attribute A to task T (A→T) and from task T to attribute A (T→A).
• The metric is decoupled from accuracy and designed to reflect real-world fairness concerns by incorporating base rates of protected attributes.
• Uses a decomposition of prediction errors to isolate those that amplify existing correlations in the training data, particularly focusing on misclassifications that reinforce or distort existing associations.
• Employs confidence intervals to account for the Rashomon Effect—where small changes in random seed lead to large fairness metric fluctuations—highlighting the instability of fairness metrics across model runs.
• Recommends against predicting sensitive attributes where possible, as this trivially eliminates T→A amplification and reduces normative ambiguity.
• Evaluates the metric across diverse settings, including computer vision (e.g., gender and painting classification) and criminal risk prediction (e.g., COMPAS), to demonstrate context-dependent validity.

Experimental results

Research questions

• RQ1 How can bias amplification be meaningfully decomposed into distinct directional pathways (A→T vs. T→A) to improve diagnostic precision?
• RQ2 To what extent do existing bias amplification metrics fail to account for base rates of protected attributes, and how does this affect fairness evaluation?
• RQ3 In settings like risk prediction where future outcomes are unknown and labels are biased, can low bias amplification still coexist with high fairness disparities?
• RQ4 How does model variability across random seeds affect the reliability of fairness metrics, and what role do confidence intervals play in mitigating this?
• RQ5 When is bias amplification an appropriate fairness metric, and when should it be avoided in favor of other fairness criteria?

Key findings

• The proposed BiasAmp→ metric successfully disentangles A→T and T→A bias amplification, enabling more targeted and normatively informed interventions.
• In the COMPAS risk prediction setting, bias amplification was near zero across all thresholds, yet significant False Positive Rate disparities persisted, demonstrating that low amplification does not imply fairness.
• Models with perfect accuracy exhibit zero bias amplification, showing that bias amplification is aligned with accuracy and does not conflict with it.
• The metric reveals that T→A amplification can be eliminated by avoiding prediction of sensitive attributes, which has strong normative implications for model design.
• Confidence intervals are essential for interpreting BiasAmp→ due to high variability across model runs, underscoring the Rashomon Effect in fairness metrics.
• Bias amplification is not a sufficient fairness metric in settings with historically biased labels or non-observable future outcomes, such as criminal recidivism prediction.