[Paper Review] Face Synthesis for Eyeglass-Robust Face Recognition

Jianzhu Guo, Xiangyu Zhu+2 more|arXiv (Cornell University)|Jun 4, 2018

Face recognition and analysis27 references4 citations

TL;DR

This paper proposes a 3D-based virtual face synthesis method to generate high-fidelity eyeglass-wearing face images for training deep learning models, significantly improving face recognition robustness to eyeglasses. By synthesizing diverse eyeglass variations using 3D face and eyeglass models, and combining them with a novel metric learning loss, the approach achieves state-of-the-art performance, especially under challenging low-far false alarm rates, with a 20% relative improvement at FAR=10^{-6} on protocol IV.

ABSTRACT

In the application of face recognition, eyeglasses could significantly degrade the recognition accuracy. A feasible method is to collect large-scale face images with eyeglasses for training deep learning methods. However, it is difficult to collect the images with and without glasses of the same identity, so that it is difficult to optimize the intra-variations caused by eyeglasses. In this paper, we propose to address this problem in a virtual synthesis manner. The high-fidelity face images with eyeglasses are synthesized based on 3D face model and 3D eyeglasses. Models based on deep learning methods are then trained on the synthesized eyeglass face dataset, achieving better performance than previous ones. Experiments on the real face database validate the effectiveness of our synthesized data for improving eyeglass face recognition performance.

Motivation & Objective

• To address the challenge of degraded face recognition accuracy due to eyeglasses, especially thick black-framed ones.
• To overcome the difficulty of collecting paired real images with and without eyeglasses for training.
• To develop a virtual data augmentation method that generates realistic, diverse eyeglass-wearing face images for robust model training.
• To improve deep learning models' generalization to intra-class variations caused by eyeglasses through synthetic data and a novel loss function.

Proposed method

• Reconstructs 3D face models from 2D images using 3DMM fitting or regression methods.
• Fits 3D eyeglasses onto reconstructed 3D faces using anchor points and optimizes alignment via Eq. (1), minimizing 3D projection error.
• Applies z-buffer and Phong illumination models for realistic rendering, blending synthetic eyeglasses onto original images.
• Introduces randomness in 3D eyeglass shapes, lighting, pitch, and vertical translation to increase diversity of synthetic data.
• Constructs a synthetic training dataset, MsCeleb-Eyeglass, by combining real MsCeleb images with synthesized eyeglass-wearing versions.
• Proposes a novel metric learning loss with gradual sampling to better utilize synthetic data and improve feature discrimination.

Experimental results

Research questions

• RQ1 Can virtual synthesis of eyeglass-wearing faces improve deep face recognition performance under real-world occlusion?
• RQ2 How effective is synthetic data augmentation using 3D face and eyeglass models in reducing performance degradation due to eyeglasses?
• RQ3 Does a novel metric learning loss with gradual sampling enhance model generalization on synthetic and real data?
• RQ4 How does the performance of models trained on synthetic data compare to real-data-only baselines under varying levels of eyeglass occlusion?

Key findings

• The ResNet-22-D model trained on a mixture of real MsCeleb and synthetic MsCeleb-Eyeglass data achieved the highest performance across all protocols, especially on protocol IV.
• On protocol IV, which features eyeglasses in both probe and gallery sets, the model achieved 90.14% TPR at FAR=10^{-4}, a 12% relative improvement over the baseline.
• At FAR=10^{-6}, the model improved recognition performance by approximately 20% compared to the baseline, demonstrating strong robustness to severe occlusion.
• The proposed synthetic data and loss function significantly enhanced performance on protocols III and IV, where eyeglass occlusion was most detrimental.
• The ROC curves in Fig. 4 show consistent superiority of models trained with synthetic data and the new loss, especially at low FAR thresholds.
• The MeGlass dataset, containing 1.7K identities, was released publicly to support future research in eyeglass-robust face recognition.