[Paper Review] Masked Face Recognition using ResNet-50

Bishwas Mandal, Adaeze Okeukwu+1 more|arXiv (Cornell University)|Apr 19, 2021

Face recognition and analysisComputer Science18 references70 citations

TL;DR

The paper fine-tunes a pre-trained ResNet-50 model for masked face recognition, achieving about 89.7% accuracy on unmasked faces and 47.9% on masked faces using RMFRD data.

ABSTRACT

Over the last twenty years, there have seen several outbreaks of different coronavirus diseases across the world. These outbreaks often led to respiratory tract diseases and have proved to be fatal sometimes. Currently, we are facing an elusive health crisis with the emergence of COVID-19 disease of the coronavirus family. One of the modes of transmission of COVID- 19 is airborne transmission. This transmission occurs as humans breathe in the droplets released by an infected person through breathing, speaking, singing, coughing, or sneezing. Hence, public health officials have mandated the use of face masks which can reduce disease transmission by 65%. For face recognition programs, commonly used for security verification purposes, the use of face mask presents an arduous challenge since these programs were typically trained with human faces devoid of masks but now due to the onset of Covid-19 pandemic, they are forced to identify faces with masks. Hence, this paper investigates the same problem by developing a deep learning based model capable of accurately identifying people with face-masks. In this paper, the authors train a ResNet-50 based architecture that performs well at recognizing masked faces. The outcome of this study could be seamlessly integrated into existing face recognition programs that are designed to detect faces for security verification purposes.

Motivation & Objective

• Investigate the challenge of recognizing identities when faces are masked due to COVID-19.
• Assess whether transfer learning with a pre-trained ResNet-50 can adapt to masked-face recognition.
• Evaluate the impact of occlusion on recognition performance and identify effective training strategies.
• Provide detailed hyperparameter tuning and architectural considerations for masked-face recognition.

Proposed method

• Apply transfer learning by fine-tuning a pre-trained ResNet-50 model on unmasked faces.
• Evaluate the fine-tuned model on masked-face data to assess recognition capability under occlusion.
• Experiment with data augmentation (random horizontal flips) and dataset preprocessing to 180x180 inputs.
• Conduct hyperparameter tuning across batch size, optimizer, dropout, learning rate, and epochs to maximize performance.
• Use RMFRD real-world masked face dataset with 70/30 train/validation split and 77 classes meeting the eight-image-per-class criterion.

Experimental results

Research questions

• RQ1 Can a ResNet-50-based model pretrained on unmasked faces accurately identify individuals when their faces are masked?
• RQ2 How does performance differ between unmasked and masked faces using transfer learning and hyperparameter tuning?
• RQ3 What data and training strategies (augmentation, cropping, domain adaptation) influence masked-face recognition performance?
• RQ4 What are the optimal hyperparameters (optimizer, batch size, dropout, learning rate, epochs) for masked-face recognition with ResNet-50?

Key findings

• Fine-tuning a pre-trained ResNet-50 on unmasked faces yields an accuracy of 89.7016% with precision 0.8993, recall 0.8970, and F1 0.897.
• On masked faces, the best result is 47.91% accuracy with precision 0.4613, recall 0.4719, and F1 0.4473.
• Data imbalance (far more unmasked than masked images) and occlusion reduce masked-face recognition performance compared to unmasked-face recognition.
• Hyperparameter tuning (switching from SGD to Adam, adjusting batch size, dropout, and learning rate) improves masked-face accuracy from ~21% to ~44.73% (reported as F1).
• The study concludes that occlusion significantly challenges CNNs for face recognition and suggests future techniques such as data augmentation with simulated masks and domain adaptation.