[Paper Review] Analysis of face detection, face landmarking, and face recognition performance with masked face images

Ožbej Golob|arXiv (Cornell University)|Jun 3, 2022

Face recognition and analysis4 citations

TL;DR

This study evaluates the impact of face masks on face detection, landmarking, and recognition using HOG and CNN detectors, 5- and 68-point landmark predictors, and a VGG16-based face recognition model. Results show significant performance degradation across all tasks due to mask occlusion, with face recognition accuracy dropping by 9.9% on masked images, though fine-tuning on masked data improved accuracy to 95.2%.

ABSTRACT

Face recognition has become an essential task in our lives. However, the current COVID-19 pandemic has led to the widespread use of face masks. The effect of wearing face masks is currently an understudied issue. The aim of this paper is to analyze face detection, face landmarking, and face recognition performance with masked face images. HOG and CNN face detectors are used for face detection in combination with 5-point and 68-point face landmark predictors and VGG16 face recognition model is used for face recognition on masked and unmasked images. We found that the performance of face detection, face landmarking, and face recognition is negatively impacted by face masks

Motivation & Objective

• To assess the impact of face masks on face detection, landmarking, and recognition performance.
• To evaluate the robustness of HOG and CNN face detectors under mask occlusion.
• To measure the degradation in 5-point and 68-point face landmarking accuracy due to masks.
• To investigate whether fine-tuning a pre-trained VGG16 model on masked images improves recognition accuracy.
• To quantify the extent of performance loss across multiple benchmark datasets with and without masks.

Proposed method

• Face detection was performed using HOG-based and CNN-based detectors implemented in Dlib.
• Face landmarking was evaluated using normalized root mean square error (NRMSE) relative to inter-ocular distance (IOD) for 5- and 68-point predictors.
• NRMSE was computed per image using the Euclidean distance between ground-truth and predicted landmark coordinates, normalized by IOD.
• Face recognition used a fine-tuned VGG16 model with the last fully connected layer replaced by a softmax classifier for classification.
• The VGG16 model was first trained on unmasked images and then retrained on masked images from the CASIA dataset.
• Performance was evaluated on four in-the-wild datasets: LFPW, HELEN, AFW, and IBUG, with masked versions generated using the MaskTheFace tool.

Experimental results

Research questions

• RQ1 How does face mask occlusion affect the accuracy of HOG and CNN-based face detectors?
• RQ2 To what extent does mask wear degrade the performance of 5-point and 68-point face landmarking models?
• RQ3 How does face recognition accuracy change when using masked images compared to unmasked images with a pre-trained VGG16 model?
• RQ4 Can fine-tuning a face recognition model on masked images mitigate performance degradation?
• RQ5 What is the relative impact of mask occlusion on different components of the face recognition pipeline?

Key findings

• HOG face detector accuracy dropped from 89.0% on unmasked to 64.7% on masked images, a 24.3 percentage point decrease on average.
• CNN face detector accuracy decreased from 98.5% to 77.8%, a 20.7 percentage point drop on average.
• 68-point landmarking NRMSE increased from 0.066 to 0.171 (nearly 3x) with the HOG detector and from 0.115 to 0.214 (nearly 2x) with the CNN detector on masked images.
• 5-point landmarking NRMSE increased from 0.060 to 0.128 (2x) with the HOG detector and from 0.067 to 0.129 (2x) with the CNN detector on masked images.
• Face recognition accuracy on masked images dropped from 96.6% (on unmasked) to 86.7% when using a VGG16 model pre-trained on unmasked data.
• Fine-tuning the VGG16 model on masked images improved recognition accuracy to 95.2% on masked images, demonstrating the effectiveness of domain-specific adaptation.