[Paper Review] Distill and De-bias: Mitigating Bias in Face Verification using Knowledge Distillation

Prithviraj Dhar, Joshua Gleason+4 more|arXiv (Cornell University)|Dec 17, 2021

Face recognition and analysis4 citations

TL;DR

This paper proposes Distill and De-bias (D&D) and D&D++, novel knowledge distillation-based methods to reduce gender and skintone bias in face verification. By transferring attention patterns from a teacher network trained on one attribute category (e.g., light skintone) to a student network trained on another (e.g., dark skintone), D&D enforces consistent spatial attention across demographic groups, significantly reducing bias while improving verification accuracy on IJB-C beyond adversarial baselines.

ABSTRACT

Face recognition networks generally demonstrate bias with respect to sensitive attributes like gender, skintone etc. For gender and skintone, we observe that the regions of the face that a network attends to vary by the category of an attribute. This might contribute to bias. Building on this intuition, we propose a novel distillation-based approach called Distill and De-bias (D&D) to enforce a network to attend to similar face regions, irrespective of the attribute category. In D&D, we train a teacher network on images from one category of an attribute; e.g. light skintone. Then distilling information from the teacher, we train a student network on images of the remaining category; e.g., dark skintone. A feature-level distillation loss constrains the student network to generate teacher-like representations. This allows the student network to attend to similar face regions for all attribute categories and enables it to reduce bias. We also propose a second distillation step on top of D&D, called D&D++. Here, we distill the `un-biasedness' of the D&D network into a new student network, the D&D++ network, while training this new network on all attribute categories; e.g., both light and dark skintones. This helps us train a network that is less biased for an attribute, while obtaining higher face verification performance than D&D. We show that D&D++ outperforms existing baselines in reducing gender and skintone bias on the IJB-C dataset, while obtaining higher face verification performance than existing adversarial de-biasing methods. We evaluate the effectiveness of our proposed methods on two state-of-the-art face recognition networks: ArcFace and Crystalface.

Motivation & Objective

• To address bias in face recognition networks related to sensitive attributes like gender and skintone.
• To investigate whether differences in attention regions across demographic groups contribute to bias.
• To develop a non-adversarial, distillation-based method that enforces consistent attention across attribute categories.
• To improve face verification performance while reducing bias, especially compared to adversarial de-biasing techniques.
• To ensure generalization across different face recognition models like ArcFace and Crystalface.

Proposed method

• Train a teacher network on images from one category of a sensitive attribute (e.g., light skintone) to learn representative attention patterns.
• Use feature-level knowledge distillation to transfer the teacher’s attention patterns to a student network trained on the other category (e.g., dark skintone), constraining the student to attend to similar face regions.
• Apply D&D++ by distilling the 'unbiased' behavior of the D&D network into a new student network trained on all attribute categories, preserving fairness and improving performance.
• Use GradCAM to visualize and validate that attention maps become more consistent across demographic groups after D&D and D&D++.
• Train and evaluate on the IJB-C dataset using ArcFace and Crystalface as backbone networks.
• Compare D&D and D&D++ against adversarial de-biasing methods (e.g., PASS, IVE) and balanced training baselines.

Experimental results

Research questions

• RQ1 Does face recognition networks' attention vary significantly across gender and skintone categories, contributing to bias?
• RQ2 Can knowledge distillation be used to align attention patterns across demographic groups to reduce bias?
• RQ3 Can a two-stage distillation process (D&D followed by D&D++) simultaneously reduce bias and improve face verification accuracy?
• RQ4 Does the proposed method generalize across different face recognition models and sensitive attributes?
• RQ5 How does D&D++ compare to adversarial de-biasing methods in terms of fairness and verification performance?

Key findings

• D&D reduces gender and skintone bias in ArcFace and Crystalface networks by aligning attention maps across demographic groups, as shown via GradCAM visualization.
• D&D++ achieves higher face verification performance than state-of-the-art adversarial de-biasing methods on the IJB-C dataset while maintaining lower bias.
• The method significantly reduces bias without requiring adversarial training or modifying the backbone network architecture.
• Feature-level distillation effectively transfers attention patterns, leading to consistent processing of faces across different skintone and gender categories.
• The approach generalizes to different face recognition models and sensitive attributes, including both gender and skintone.
• D&D++ outperforms balanced training baselines and other non-adversarial methods like GAC and IVE in fairness and accuracy trade-offs.