[Paper Review] Masked Face Recognition with Generative-to-Discriminative Representations

Shiming Ge, Weijia Guo+4 more|arXiv (Cornell University)|May 27, 2024

Face recognition and analysis4 citations

TL;DR

This paper proposes a generative-to-discriminative representation learning framework for masked face recognition, using a three-module network: a pretrained generative encoder for occlusion-robust, category-aware descriptors, a discriminative reformer fine-tuned via knowledge distillation to produce identity-aware features, and a classifier head. The method achieves state-of-the-art performance on synthetic and real masked face benchmarks, with 92.86% accuracy on CPLFW and 98.02% on synthetic LFW using ArcFace as a teacher.

ABSTRACT

Masked face recognition is important for social good but challenged by diverse occlusions that cause insufficient or inaccurate representations. In this work, we propose a unified deep network to learn generative-to-discriminative representations for facilitating masked face recognition. To this end, we split the network into three modules and learn them on synthetic masked faces in a greedy module-wise pretraining manner. First, we leverage a generative encoder pretrained for face inpainting and finetune it to represent masked faces into category-aware descriptors. Attribute to the generative encoder's ability in recovering context information, the resulting descriptors can provide occlusion-robust representations for masked faces, mitigating the effect of diverse masks. Then, we incorporate a multi-layer convolutional network as a discriminative reformer and learn it to convert the category-aware descriptors into identity-aware vectors, where the learning is effectively supervised by distilling relation knowledge from off-the-shelf face recognition model. In this way, the discriminative reformer together with the generative encoder serves as the pretrained backbone, providing general and discriminative representations towards masked faces. Finally, we cascade one fully-connected layer following by one softmax layer into a feature classifier and finetune it to identify the reformed identity-aware vectors. Extensive experiments on synthetic and realistic datasets demonstrate the effectiveness of our approach in recognizing masked faces.

Motivation & Objective

• Address the challenge of masked face recognition due to occlusion and information loss in real-world applications.
• Overcome limitations of purely generative or discriminative approaches by combining their strengths: generative models for context recovery and discriminative models for identity discrimination.
• Develop a unified, end-to-end trainable framework that maintains performance on both masked and unmasked faces.
• Improve robustness across diverse mask types by learning consistent, identity-aware representations despite semantic variations in occlusions.
• Enable practical deployment through efficient inference speed (23.35 FPS on RTX 3090).

Proposed method

• Utilizes a pretrained face inpainting encoder as the generative encoder, fine-tuned to produce category-aware descriptors from masked faces, leveraging its ability to recover contextual facial information.
• Introduces a 22-layer convolutional discriminative reformer that transforms category-aware descriptors into identity-aware features using knowledge distillation from a pretrained face recognition model (e.g., ArcFace or VGGFace2).
• Employs a multi-loss training strategy: cross-entropy loss for identity classification, L1 loss for feature reconstruction, and knowledge distillation to transfer relation-aware representations from a teacher model.
• Performs greedy, module-wise pretraining: first pretrain the generative encoder on synthetic masked faces, then jointly fine-tune the discriminative reformer and feature classifier using distillation and classification signals.
• Freezes the backbone (generative encoder + discriminative reformer) after pretraining and finetunes only the final classifier head on downstream recognition tasks.
• Uses data augmentation and synthetic masked face generation to enhance robustness and generalization across diverse mask types.

Experimental results

Research questions

• RQ1 Can a unified network architecture combining generative and discriminative components improve masked face recognition accuracy compared to standalone generative or discriminative models?
• RQ2 Does knowledge distillation from a high-accuracy face recognizer enhance the discriminative power of representations learned from masked faces?
• RQ3 Can a generative encoder pre-trained on face inpainting produce robust, occlusion-invariant descriptors that generalize to masked face recognition?
• RQ4 How does the proposed generative-to-discriminative pipeline perform on both synthetic and realistic masked face datasets?
• RQ5 To what extent does the method preserve performance on unmasked face recognition while improving masked face recognition?

Key findings

• The proposed G2D model achieves 92.86% accuracy on the CPLFW benchmark for masked face recognition, outperforming prior methods including MaskInv and SphereFace.
• On synthetic masked LFW, the model reaches 98.02% verification accuracy when using ArcFace as the distillation teacher, compared to 97.58% with VGGFace2.
• The similarity score distribution of the learned representations shows minimal overlap between positive and negative pairs, indicating strong discriminability close to the ideal case.
• The model maintains competitive performance on normal face recognition, with 92.23% accuracy on CPLFW, showing robustness across masked and unmasked inputs.
• Inference speed reaches 23.35 FPS on a single RTX 3090 GPU, demonstrating feasibility for real-time deployment in urban governance and public safety systems.
• Ablation studies confirm that knowledge distillation is more effective than direct cross-entropy supervision for identity learning, as models trained with distillation show better convergence and performance.