[Paper Review] Masked Face Recognition for Secure Authentication

Aqeel Anwar, Arijit Raychowdhury|arXiv (Cornell University)|Aug 25, 2020

Face recognition and analysis15 references147 citations

TL;DR

The paper introduces MaskTheFace, an open-source tool to generate masked-face datasets, and retrains Facenet to improve masked-face recognition by about 38% TP rate, with robust performance on LFW-SM and real-world MFR2 data.

ABSTRACT

With the recent world-wide COVID-19 pandemic, using face masks have become an important part of our lives. People are encouraged to cover their faces when in public area to avoid the spread of infection. The use of these face masks has raised a serious question on the accuracy of the facial recognition system used for tracking school/office attendance and to unlock phones. Many organizations use facial recognition as a means of authentication and have already developed the necessary datasets in-house to be able to deploy such a system. Unfortunately, masked faces make it difficult to be detected and recognized, thereby threatening to make the in-house datasets invalid and making such facial recognition systems inoperable. This paper addresses a methodology to use the current facial datasets by augmenting it with tools that enable masked faces to be recognized with low false-positive rates and high overall accuracy, without requiring the user dataset to be recreated by taking new pictures for authentication. We present an open-source tool, MaskTheFace to mask faces effectively creating a large dataset of masked faces. The dataset generated with this tool is then used towards training an effective facial recognition system with target accuracy for masked faces. We report an increase of 38% in the true positive rate for the Facenet system. We also test the accuracy of re-trained system on a custom real-world dataset MFR2 and report similar accuracy.

Motivation & Objective

• Address the degradation of face recognition accuracy when faces are masked due to COVID-19.
• Enable use of existing unmasked datasets by augmenting them with masked variants without re-collecting data.
• Develop and release MaskTheFace to create masked-face datasets from existing datasets.
• Evaluate masked-face recognition performance using Facenet with simulated and real masked datasets.

Proposed method

• Develop MaskTheFace to mask faces using dlib-based landmarks to fit 5 mask types across 24 patterns and various angles.
• Generate masked-face datasets from existing datasets (e.g., VGGFace2 mini) without collecting new photos.
• Train Facenet embeddings (Inception-ResNet-v1, embedding size 512) on masked and unmasked data.
• Evaluate performance using Max Accuracy, ACC@FAR=0.1%, and TPR@FAR=0.1% metrics across multiple test sets.
• Compare networks trained on unmasked data versus MaskTheFace-augmented data to assess robustness across masked/unmasked pairs.

Experimental results

Research questions

• RQ1 Can masked faces be effectively used to train and improve a state-of-the-art face recognition system without re-collecting data?
• RQ2 How does a network trained on simulated masked faces perform on real-world masked-face data?
• RQ3 What is the impact of masking on recognition performance metrics (accuracy, FAR, TPR) for masked versus unmasked training regimes?

Key findings

• MaskTheFace-enabled retraining yields a ~38% increase in true positive rate for Facenet on masked and unmasked faces.
• The retrained mask-network achieves comparable or slightly better performance than the no-mask network on several cross-dataset pairs.
• On LFW-SM, the mask-network maintains robust accuracy across mask types, reducing TPR@FAR=0.1% degradation compared to the no-mask network.
• On the real-world MFR2 dataset, the mask-network shows ~34% (TPR@FAR=0.2%), ~17% (Accuracy@FAR=0.2%), and ~6% (Max accuracy) improvements over the no-mask network.
• The masked-network performance on MFR2 remains close to the LFW-SM results, indicating robustness to real-world masked faces.