[Paper Review] Masked Face Recognition Challenge: The InsightFace Track Report

Jiankang Deng, Jia Guo+3 more|arXiv (Cornell University)|Aug 18, 2021

Face recognition and analysis37 references10 citations

TL;DR

This paper introduces a large-scale, privacy-preserving benchmark for masked face recognition, featuring three distinct test sets—masked, children, and multi-racial—using real-world data to evaluate deep face recognition models. It reports significant performance gains, with top models achieving 88.97% accuracy on the masked test set and 93.51% on the MR-All set, demonstrating the effectiveness of occlusion-robust feature learning without external data or pre-trained models.

ABSTRACT

During the COVID-19 coronavirus epidemic, almost everyone wears a facial mask, which poses a huge challenge to deep face recognition. In this workshop, we organize Masked Face Recognition (MFR) challenge and focus on bench-marking deep face recognition methods under the existence of facial masks. In the MFR challenge, there are two main tracks: the InsightFace track and the WebFace260M track. For the InsightFace track, we manually collect a large-scale masked face test set with 7K identities. In addition, we also collect a children test set including 14K identities and a multi-racial test set containing 242K identities. By using these three test sets, we build up an online model testing system, which can give a comprehensive evaluation of face recognition models. To avoid data privacy problems, no test image is released to the public. As the challenge is still under-going, we will keep on updating the top-ranked solutions as well as this report on the arxiv.

Motivation & Objective

• To address the challenge of degraded performance in deep face recognition systems due to facial masks during the COVID-19 pandemic.
• To establish a fair, comprehensive, and privacy-preserving evaluation benchmark for masked face recognition.
• To enable reproducible and comparable model evaluation across different training data and model complexity constraints.
• To promote fairness in face recognition by including diverse racial groups in the benchmark.

Proposed method

• The authors collected a real-world masked face test set with 6,964 identities and 20,892 images, ensuring privacy by not releasing test images.
• They created a children test set with 14,344 identities and a multi-racial test set with 242,000 identities, covering African, Caucasian, South Asian, and East Asian populations.
• An online model testing system was built to evaluate submissions without exposing test data, ensuring fairness and privacy.
• Participants used fixed training datasets (MS1M and Glint360K) with no external data or pre-trained models allowed, ensuring reproducibility.
• The evaluation protocol included multiple test sets and strict constraints on model size and computational complexity.
• Data augmentation with facial masks was permitted, but must be reproducible to ensure consistent training across submissions.

(a) Masked Test Set

Experimental results

Research questions

• RQ1 How effective are deep face recognition models when applied to masked faces compared to non-masked faces?
• RQ2 What performance gains can be achieved through occlusion-robust feature learning without using external data or pre-trained models?
• RQ3 How does model performance vary across different demographic groups, particularly in multi-racial and children test sets?
• RQ4 To what extent can data augmentation with masks improve recognition accuracy under real-world occlusion conditions?
• RQ5 How do different training data scales (MS1M vs. Glint360K) affect the robustness of face recognition models to facial masks?

Key findings

• The best model on the MS1M sub-track achieved 84.169% accuracy on the masked test set and 90.452% on the MR-All test set, showing a 7% absolute improvement over baseline models.
• The best model on the Glint360K sub-track achieved 88.972% accuracy on the masked test set and 93.512% on the MR-All test set, representing a 6% absolute improvement over baseline.
• Top-performing models demonstrated strong generalization across diverse demographics, including children and multi-racial groups, indicating improved fairness.
• The use of masked face augmentation in training led to significant performance gains, confirming its effectiveness in enhancing occlusion robustness.
• The benchmark successfully enabled fair and reproducible evaluation across global participants, with 123 valid submissions for the MS1M sub-track and 69 for the Glint360K sub-track.
• The authors confirmed the validity of top-ranked models through retraining, ensuring the reliability of the leaderboard results.

(b) Children Test Set