[Paper Review] Masked Face Recognition Challenge: The WebFace260M Track Report

Zheng Zhu, Guan Huang+10 more|arXiv (Cornell University)|Aug 16, 2021

Face recognition and analysis24 references24 citations

TL;DR

This paper reports the WebFace260M Track of the Masked Face Recognition Challenge, detailing the dataset, FRUITS evaluation protocol, test sets, submission rules, baseline, and first-phase results for masked and non-masked face recognition under real-world conditions.

ABSTRACT

According to WHO statistics, there are more than 204,617,027 confirmed COVID-19 cases including 4,323,247 deaths worldwide till August 12, 2021. During the coronavirus epidemic, almost everyone wears a facial mask. Traditionally, face recognition approaches process mostly non-occluded faces, which include primary facial features such as the eyes, nose, and mouth. Removing the mask for authentication in airports or laboratories will increase the risk of virus infection, posing a huge challenge to current face recognition systems. Due to the sudden outbreak of the epidemic, there are yet no publicly available real-world masked face recognition (MFR) benchmark. To cope with the above-mentioned issue, we organize the Face Bio-metrics under COVID Workshop and Masked Face Recognition Challenge in ICCV 2021. Enabled by the ultra-large-scale WebFace260M benchmark and the Face Recognition Under Inference Time conStraint (FRUITS) protocol, this challenge (WebFace260M Track) aims to push the frontiers of practical MFR. Since public evaluation sets are mostly saturated or contain noise, a new test set is gathered consisting of elaborated 2,478 celebrities and 60,926 faces. Meanwhile, we collect the world-largest real-world masked test set. In the first phase of WebFace260M Track, 69 teams (total 833 solutions) participate in the challenge and 49 teams exceed the performance of our baseline. There are second phase of the challenge till October 1, 2021 and on-going leaderboard. We will actively update this report in the future.

Motivation & Objective

• Motivate robust masked face recognition (MFR) under real-world mask usage amid COVID-19.
• Leverage the WebFace260M dataset and FRUITS protocol to evaluate MFR under inference-time constraints.
• Construct a large, realistic test set for SFR and MFR across controlled, wild, and cross-age scenarios.
• Provide a baseline solution and baseline performance to guide future research in practical MFR.

Proposed method

• Adopt FRUITS protocol to measure inference-time constrained performance on a single CPU core (1000 ms limit).
• Create a new test set with 2,478 identities and 60,926 masked faces, plus detailed SFR/test-set attributes (Controlled, Wild, Cross-age).
• Use 1:1 verification with FNMR at fixed FMR thresholds to compare masked and non-masked performance across settings (All, Wild, Controlled).
• Balance MFR and SFR metrics via a weighted final score (All (MFR&SFR) = 0.25 All-Masked + 0.75 All(SFR)).
• Provide a reproducible baseline model (ResNet-50 with ArcFace loss) and a Gluon-based codebase for fair comparison.

Experimental results

Research questions

• RQ1 How does real-world masked face recognition perform under practical inference-time constraints compare to non-masked face recognition?
• RQ2 What is the impact of mask-wearing across different test conditions (Controlled, Wild, Cross-age) on MFR and SFR performance?
• RQ3 Can large-scale, publicly available training data (WebFace260M) close the performance gap between academia and industry for MFR?
• RQ4 What is the relative contribution of face detection/alignment costs to overall latency under FRUITS constraints?

Key findings

• First-phase participation: 69 teams submitted 833 solutions; 49 teams exceeded the baseline.
• All (MFR&SFR) metric ranking top result 0.0980; Controlled (MFR&SFR) 0.0500; Wild (MFR&SFR) 0.1222; All (SFR) 0.0162; Wild (SFR) 0.0270; Controlled (SFR) 0.0018.
• Baseline model (ResNet-50 with ArcFace, WebFace12M) achieved FNMR@FMR=1e-5 as 0.2342 (All MFR&SFR), 0.2816 (Wild MFR&SFR), 0.1483 (Controlled MFR&SFR), and 0.0871 (All SFR).
• Test-set statistics include 2,478 identities and 57,715 SFR faces, plus 3,211 masked faces from 862 identities (All-Masked).
• Test-time and recognition-time costs for the baseline: Detection 157 ms, Recognition 172 ms, Total 329 ms.
• Reset and ranking criteria combine MFR and SFR scores to mitigate overfitting to masked or non-masked data.