[Paper Review] My Eyes Are Up Here: Promoting Focus on Uncovered Regions in Masked Face Recognition

Pedro C. Neto, Fadi Boutros+5 more|arXiv (Cornell University)|Aug 2, 2021

Face recognition and analysis22 references19 citations

TL;DR

This paper proposes a novel training paradigm that enhances masked face recognition (MFR) by focusing on uncovered facial regions through a modified triplet loss combining mean squared error (MSE) with traditional triplet loss. The method improves verification performance in masked-vs-unmasked comparisons by guiding embeddings to prioritize unmasked areas, achieving state-of-the-art results on both synthetic and real masked face datasets with AUC up to 0.985 and EER reduced to 6.2%.

ABSTRACT

The recent Covid-19 pandemic and the fact that wearing masks in public is now mandatory in several countries, created challenges in the use of face recognition systems (FRS). In this work, we address the challenge of masked face recognition (MFR) and focus on evaluating the verification performance in FRS when verifying masked vs unmasked faces compared to verifying only unmasked faces. We propose a methodology that combines the traditional triplet loss and the mean squared error (MSE) intending to improve the robustness of an MFR system in the masked-unmasked comparison mode. The results obtained by our proposed method show improvements in a detailed step-wise ablation study. The conducted study showed significant performance gains induced by our proposed training paradigm and modified triplet loss on two evaluation databases.

Motivation & Objective

• Address the challenge of masked face recognition (MFR) due to pandemic-related mask-wearing.
• Improve verification performance in masked-vs-unmasked face comparisons, a critical real-world scenario.
• Develop a training paradigm that enhances model robustness by focusing on uncovered facial regions.
• Investigate the impact of loss function modifications on MFR performance through ablation studies.
• Demonstrate effectiveness on both synthetic and real-world masked face datasets.

Proposed method

• Proposes a cascaded training paradigm: first, a classification head is trained using cross-entropy (CE) loss.
• Then, the network is fine-tuned by freezing all layers except the final embedding layer.
• Introduces a modified triplet loss that incorporates mean squared error (MSE) to guide embedding learning.
• The modified loss penalizes the model if the distance between anchor and negative is smaller than anchor and positive, with a margin of 0.
• Uses Smooth Grad-CAM++ to visualize feature importance and validate focus on uncovered facial regions.
• The MSE component encourages the model to learn representations that emphasize unmasked facial areas.

Experimental results

Research questions

• RQ1 How does combining MSE with triplet loss improve MFR performance in masked-vs-unmasked verification?
• RQ2 To what extent does focusing on uncovered facial regions enhance model robustness in MFR?
• RQ3 Does a two-stage training approach (classification followed by embedding fine-tuning) improve MFR performance?
• RQ4 How does the proposed method compare to baseline models like VGGFace2 in real-world masked face recognition?
• RQ5 What is the contribution of each component (CE, triplet loss, MSE) to the final performance, as shown in ablation studies?

Key findings

• On the synthetic masked face dataset, the proposed method (CE + TL + MSE) achieved an AUC of 0.985, EER of 6.2%, and FMR10 of 4.1%.
• On the real masked face dataset, the method achieved an AUC of 0.957 in the U-M mode (unmasked vs masked), with EER reduced to 9.799%.
• The ablation study confirmed that adding MSE to the triplet loss significantly improved performance over CE + TL alone.
• The model trained only on synthetic data showed strong generalization to real masked data, indicating robustness of the proposed loss.
• Grad-CAM++ visualization confirmed that the MSE-modified model focuses more on uncovered facial regions than baseline models.
• The method did not improve performance in M-M (masked vs masked) mode, as expected, since the training objective was specifically tuned for U-M verification.