[Paper Review] Is Face Recognition Sexist? No, Gendered Hairstyles and Biology Are

Vítor Albiero, Kevin W. Bowyer|arXiv (Cornell University)|Aug 16, 2020

Face recognition and analysis18 references4 citations

TL;DR

This paper investigates why face recognition systems exhibit lower accuracy for females, identifying gendered hairstyles and biological differences in facial variability as the primary causes. Using controlled datasets and PCA analysis, the authors show that occlusion from long hair increases false non-match rates for women, while inherently smaller facial variation among women leads to higher false match rates—challenging the assumption that bias stems from training data or algorithms.

ABSTRACT

Recent news articles have accused face recognition of being "biased", "sexist" or "racist". There is consensus in the research literature that face recognition accuracy is lower for females, who often have both a higher false match rate and a higher false non-match rate. However, there is little published research aimed at identifying the cause of lower accuracy for females. For instance, the 2019 Face Recognition Vendor Test that documents lower female accuracy across a broad range of algorithms and datasets also lists "Analyze cause and effect" under the heading "What we did not do". We present the first experimental analysis to identify major causes of lower face recognition accuracy for females on datasets where previous research has observed this result. Controlling for equal amount of visible face in the test images reverses the apparent higher false non-match rate for females. Also, principal component analysis indicates that images of two different females are inherently more similar than of two different males, potentially accounting for a difference in false match rates.

Motivation & Objective

• To identify the root causes of lower face recognition accuracy for females, countering claims of algorithmic sexism.
• To investigate whether occlusion from gendered hairstyles contributes to higher false non-match rates in females.
• To examine whether inherent biological differences in facial variability between males and females affect false match rates.
• To provide a transparent, reproducible analysis using state-of-the-art deep CNN matchers and publicly available datasets.
• To challenge the assumption that training data imbalance or algorithmic design is the main cause of gender-based accuracy disparities.

Proposed method

• Controlled experiments were conducted on datasets with equal visible face area to isolate the effect of hairstyle-related occlusion.
• Principal component analysis (PCA) was used to compare the intrinsic variability of male and female face spaces.
• A state-of-the-art deep CNN matcher (ArcFace) was applied to ensure results were not dependent on a single algorithm.
• The study analyzed impostor and genuine score distributions across multiple datasets, including MORPH, Notre Dame, and VGGFace2.
• Face visibility percentages and face difference heatmaps were computed to quantify occlusion effects.
• The PCA inversion experiment was performed to assess whether male face variability could be adjusted to match female face variability.

Experimental results

Research questions

• RQ1 Does the higher false non-match rate for females result from greater facial occlusion due to gendered hairstyles?
• RQ2 To what extent do biological differences in facial variability between males and females contribute to higher false match rates for females?
• RQ3 Can controlling for visible face area eliminate the observed gender disparity in false non-match rates?
• RQ4 Is the difference in impostor score distributions between males and females primarily due to inherent facial similarity differences?
• RQ5 Does the observed gender bias in face recognition accuracy persist when training and test data are gender-balanced?

Key findings

• Controlling for equal visible face area in test images reversed the apparent higher false non-match rate for females, indicating that occlusion from long hair is a major contributor.
• PCA analysis revealed that, on average, two different female faces are more similar than two different male faces, explaining the higher false match rate for females.
• The impostor distribution for females is naturally centered at higher similarity values due to reduced facial variability, which cannot be corrected by standard matching algorithms.
• Even when reducing the number of female images to balance data size, the PCA curve for females remained less variable than for males, confirming biological differences as a root cause.
• The study found no evidence that training data imbalance or algorithmic design is the primary cause of gender-based accuracy disparities.
• The results suggest that face recognition systems are not inherently sexist, but the observed gender bias arises from real biological and sociological factors such as hairstyles and facial morphology.