[Paper Review] Mitigate Bias in Face Recognition using Skewness-Aware Reinforcement Learning

Mei Wang, Weihong Deng|arXiv (Cornell University)|Nov 25, 2019

Face recognition and analysisComputer Science45 references36 citations

TL;DR

The paper introduces RL-RBN, a reinforcement learning framework that learns adaptive margins for non-Caucasian races to mitigate bias in face recognition, and releases ethnicity-aware datasets for fairness study.

ABSTRACT

Racial equality is an important theme of international human rights law, but it has been largely obscured when the overall face recognition accuracy is pursued blindly. More facts indicate racial bias indeed degrades the fairness of recognition system and the error rates on non-Caucasians are usually much higher than Caucasians. To encourage fairness, we introduce the idea of adaptive margin to learn balanced performance for different races based on large margin losses. A reinforcement learning based race balance network (RL-RBN) is proposed. We formulate the process of finding the optimal margins for non-Caucasians as a Markov decision process and employ deep Q-learning to learn policies for an agent to select appropriate margin by approximating the Q-value function. Guided by the agent, the skewness of feature scatter between races can be reduced. Besides, we provide two ethnicity aware training datasets, called BUPT-Globalface and BUPT-Balancedface dataset, which can be utilized to study racial bias from both data and algorithm aspects. Extensive experiments on RFW database show that RL-RBN successfully mitigates racial bias and learns more balanced performance for different races.

Motivation & Objective

• Motivate fairness in face recognition by addressing racial bias that manifests even with balanced data and strong accuracy.
• Propose an adaptive-margin learning framework to balance intra- and inter-class feature separability across races.
• Investigate how reinforcement learning can optimize race-specific margins to reduce skewness between races.
• Provide ethnicity-aware training datasets to study data and algorithm biases in FR systems.

Proposed method

• Model racial bias as a Markov Decision Process and use deep Q-learning to learn adaptive margins per race.
• Define the adaptive-margin loss where Caucasians retain a fixed margin and non-Caucasian margins are policy-driven (α_j(t)).
• Use an off-line sampling phase to collect state-transition-reward tuples for training the DQN.
• Define state s as a combination of race group G, current margin M, and inter-class skewness B_inter; actions modify margins; rewards penalize intra/inter-class skewness (B_intra + B_inter).
• Train a recognition network under guidance of the learned adaptive-margin policy to achieve balanced performance across races.
• Introduce two ethnicity-aware datasets (BUPT-Globalface and BUPT-Balancedface) and evaluate on the RFW benchmark.

Experimental results

Research questions

• RQ1 Can adaptive, race-specific margins learned via reinforcement learning reduce cross-race performance skew in face recognition?
• RQ2 Does incorporating skewness-based rewards in a DQN lead to more balanced intra- and inter-class separability across races?
• RQ3 Do ethnicity-aware training datasets help in diagnosing and mitigating data- and algorithm-induced biases in FR systems?
• RQ4 How does RL-RBN compare to fixed-margin approaches (Softmax, Cosface, ArcFace) on fairness metrics across races?

Key findings

• RL-RBN and its variants substantially improve fairness metrics (lower STD and SER) compared to baseline margins.
• On RFW, RL-RBN(soft) achieves STD 0.66 and SER 1.17, versus Norm-Softmax STD 2.64 and SER 1.53 under similar settings.
• With balanced training data, RL-RBN(Cos/Arc) yield up to balanced average accuracy and reduced inter-race skew compared to fixed-margin counterparts.
• On BUPT-Globalface, RL-RBN(soft) achieves Avg 94.45 with STD 0.44 and SER 1.20, outperforming Softmax (STD 2.48, SER 1.78) and other baselines.
• On BUPT-Balancedface, RL-RBN(Cos/Arc) reach Avg around 95.0 with notably lower STD/SER than fixed-margin approaches.
• The adaptive margins tend to assign larger margins to more challenging groups (Asians, Africans) and smaller margins to easier groups, guided by the learned policy.