[Paper Review] Mitigating Gender Bias in Face Recognition Using the von Mises-Fisher Mixture Model

Jean-Rémy Conti, Nathan Noiry+3 more|arXiv (Cornell University)|Oct 24, 2022

Face recognition and analysis4 citations

TL;DR

This paper proposes a post-processing method called the Ethical Module that mitigates gender bias in pre-trained face recognition models using a fair von Mises-Fisher mixture model. By training a shallow neural network to adjust deep embeddings with gender-specific variance hyperparameters, the method significantly reduces both BFAR and BFRR metrics—achieving the best simultaneous fairness improvement at FAR=10⁻⁴ across multiple models including ArcFace, AdaCos, CosFace, and CurricularFace.

ABSTRACT

In spite of the high performance and reliability of deep learning algorithms in a wide range of everyday applications, many investigations tend to show that a lot of models exhibit biases, discriminating against specific subgroups of the population (e.g. gender, ethnicity). This urges the practitioner to develop fair systems with a uniform/comparable performance across sensitive groups. In this work, we investigate the gender bias of deep Face Recognition networks. In order to measure this bias, we introduce two new metrics, $\mathrm{BFAR}$ and $\mathrm{BFRR}$, that better reflect the inherent deployment needs of Face Recognition systems. Motivated by geometric considerations, we mitigate gender bias through a new post-processing methodology which transforms the deep embeddings of a pre-trained model to give more representation power to discriminated subgroups. It consists in training a shallow neural network by minimizing a Fair von Mises-Fisher loss whose hyperparameters account for the intra-class variance of each gender. Interestingly, we empirically observe that these hyperparameters are correlated with our fairness metrics. In fact, extensive numerical experiments on a variety of datasets show that a careful selection significantly reduces gender bias. The code used for the experiments can be found at https://github.com/JRConti/EthicalModule_vMF.

Motivation & Objective

• To address gender bias in state-of-the-art face recognition models, which exhibit high disparities in false acceptance and rejection rates between males and females.
• To develop fairness metrics—BFAR and BFRR—that reflect real-world deployment constraints, particularly security requirements and equitable performance across genders.
• To design a post-processing solution that corrects bias in pre-trained models without retraining, preserving their high accuracy while improving fairness.
• To leverage the geometric structure of deep face embeddings on a hypersphere to model intra-class variance differences between genders.
• To validate the method across multiple pre-trained models (ArcFace, AdaCos, CosFace, CurricularFace) and datasets, demonstrating robustness and generalization.

Proposed method

• Proposes a novel Fair von Mises-Fisher loss that incorporates gender-specific hyperparameters for intra-class variance, enabling the model to adapt to subgroup-specific embedding distributions.
• Trains a shallow multi-layer perceptron (Ethical Module) on top of a frozen pre-trained face recognition model to reweight embeddings based on gender-aware variance parameters.
• Uses maximum likelihood estimation to train the Ethical Module by minimizing the Fair von Mises-Fisher loss, which models embeddings as a mixture of von Mises-Fisher distributions on a hypersphere.
• Employs three variants of the Ethical Module: EM-FAR (optimizes for BFAR), EM-FRR (optimizes for BFRR), and EM-C (jointly optimizes both fairness metrics) to balance fairness and performance.
• Calibrates the model at a fixed false acceptance rate (FAR) level to ensure security constraints are met while measuring fairness disparities.
• Validates the method on LFW and MS1MV3 datasets using multiple pre-trained models, comparing fairness and accuracy across different FAR thresholds.

Experimental results

Research questions

• RQ1 Can new fairness metrics—BFAR and BFRR—better reflect the operational needs of real-world face recognition systems than existing fairness definitions?
• RQ2 To what extent can gender bias in pre-trained face recognition models be mitigated through post-processing without retraining?
• RQ3 How does modeling intra-class variance differences between genders via a von Mises-Fisher mixture model improve fairness in embedding space?
• RQ4 Can a single post-processing module simultaneously reduce both BFAR and BFRR disparities across diverse pre-trained models?
• RQ5 Is the proposed method robust to changes in the backbone model and loss function used in the pre-trained system?

Key findings

• The Ethical Module achieves the best reduction in both BFAR and BFRR simultaneously at FAR=10⁻⁴, with EM-C variant outperforming the original ArcFace model in fairness while maintaining competitive accuracy.
• At FAR=10⁻⁴, EM-C reduces BFRR from 7.28 to 4.51 and BFAR from 1.08 to 1.57 across AdaCos, outperforming both the original model and PASS-g baseline.
• For CosFace with MobileFaceNet, EM-C reduces BFRR from 8.18 to 2.71 and BFAR from 1.74 to 1.13, demonstrating strong fairness gains without significant accuracy loss.
• The EM-FAR variant achieves the lowest BFAR (1.01 at FAR=10⁻³) across all models, showing effectiveness in minimizing false acceptance disparities.
• The EM-FRR variant achieves the best BFRR (2.71) for AdaCos at FAR=10⁻⁴, indicating strong mitigation of false rejection bias.
• The method generalizes across multiple pre-trained models (ArcFace, AdaCos, CosFace, CurricularFace) and backbones (ResNet50, MobileFaceNet), proving robustness and transferability.