[Paper Review] Bias Behind the Wheel: Fairness Testing of Autonomous Driving Systems

Xinyue Li, Zhenpeng Chen+4 more|arXiv (Cornell University)|Aug 5, 2023

Traffic and Road SafetyEngineering3 citations

TL;DR

This study conducts the first comprehensive fairness evaluation of eight state-of-the-art pedestrian detectors across demographic groups using real-world datasets with manually annotated gender, age, and skin tone labels. It reveals significant accuracy disparities—19.67% lower detection for children and 7.52% for dark-skin individuals—worsened under low-brightness and low-contrast conditions, highlighting critical bias in autonomous driving systems.

ABSTRACT

This paper conducts fairness testing of automated pedestrian detection, a crucial but under-explored issue in autonomous driving systems. We evaluate eight state-of-the-art deep learning-based pedestrian detectors across demographic groups on large-scale real-world datasets. To enable thorough fairness testing, we provide extensive annotations for the datasets, resulting in 8,311 images with 16,070 gender labels, 20,115 age labels, and 3,513 skin tone labels. Our findings reveal significant fairness issues, particularly related to age. The proportion of undetected children is 20.14% higher compared to adults. Furthermore, we explore how various driving scenarios affect the fairness of pedestrian detectors. We find that pedestrian detectors demonstrate significant gender biases during night time, potentially exacerbating the prevalent societal issue of female safety concerns during nighttime out. Moreover, we observe that pedestrian detectors can demonstrate both enhanced fairness and superior performance under specific driving conditions, which challenges the fairness-performance trade-off theory widely acknowledged in the fairness literature. We publicly release the code, data, and results to support future research on fairness in autonomous driving.

Motivation & Objective

• To investigate group fairness issues in state-of-the-art pedestrian detectors across demographic groups such as gender, age, and skin tone.
• To address the lack of annotated demographic data in existing real-world datasets used for autonomous driving testing.
• To evaluate how common driving scenarios (e.g., low brightness, low contrast) affect fairness in pedestrian detection.
• To provide publicly available, large-scale annotated datasets and code to support future fairness research in autonomous driving.
• To inform researchers, engineers, and policymakers about critical bias risks in current autonomous vehicle systems.

Proposed method

• Manually annotated 8,311 real-world images from four widely used datasets with gender (16,070), age (20,115), and skin tone (3,513) labels.
• Evaluated eight deep learning-based pedestrian detectors using fairness metrics: Statistical Parity Difference (SPD) and Equal Opportunity Difference (EOD).
• Applied standard pedestrian detection evaluation using miss rate as the primary performance metric, aligned with existing literature.
• Conducted scenario-based analysis under varying lighting and contrast conditions (e.g., day vs. night) to assess fairness degradation.
• Used EOD as the primary fairness measure due to its alignment with miss rate and consistency with current detection research.
• Released a comprehensive replication package including code, data, and results to ensure reproducibility and enable future research.

Figure 1. RQ1: Miss rates for female and male pedestrians in each dataset. Statistically significant gender unfairness (i.e., miss rate difference) is indicated by labeled miss rate values. We find that pedestrian detectors tend to have poorer performance in detecting females compared to males withi

Experimental results

Research questions

• RQ1 How do state-of-the-art pedestrian detectors perform across different demographic groups in terms of fairness?
• RQ2 To what extent do lighting and contrast conditions exacerbate fairness disparities in pedestrian detection?
• RQ3 How do detection performance differences vary by age, skin tone, and gender across real-world driving scenarios?
• RQ4 What are the key factors contributing to bias in current pedestrian detection systems, particularly for children and dark-skin individuals?
• RQ5 Can the inclusion of demographic annotations in real-world datasets enable more rigorous fairness testing in autonomous driving systems?

Key findings

• Pedestrian detectors exhibit a 19.67% higher undetected proportion for children compared to adults across all datasets.
• The undetected proportion for dark-skin individuals is 7.52% higher than for light-skin individuals, indicating significant skin tone bias.
• Gender-based detection disparity is minimal, with only a 1.1% difference in undetected proportions between males and females.
• Under low-brightness and low-contrast conditions, the fairness gap for dark-skin individuals widens, increasing the undetected proportion difference from 7.14% to 9.86% between day and night scenarios.
• One-stage detectors show particularly strong bias against dark-skin pedestrians, suggesting architecture-specific fairness challenges.
• The study confirms that fairness metrics like EOD and SPD yield consistent results when applied to pedestrian detection, validating the use of EOD for fairness evaluation in this context.

Figure 2. RQ1: Miss rates for children and adults in each dataset. Statistically significant age unfairness (i.e., miss rate difference) is indicated by labeled miss rate values. The miss rate of children is consistently higher than that of adults for all datasets and all detectors.