[Paper Review] To Frontalize or Not To Frontalize: A Study of Face Pre-Processing Techniques and Their Impact on Recognition

Sandipan Banerjee, Joel Brogan+6 more|arXiv (Cornell University)|Oct 16, 2016

Face recognition and analysis26 references1 citations

TL;DR

This paper evaluates the impact of face pre-processing techniques—particularly frontalization—on deep learning-based face recognition. It proposes a novel single-image frontalization method that outperforms existing approaches, demonstrating that pre-processing for pose normalization significantly improves recognition accuracy on PaSC and Multi-PIE datasets when using CNNs.

ABSTRACT

Face recognition performance has improved remarkably in the last decade. Much of this success can be attributed to the development of deep learning techniques such as convolutional neural networks (CNNs). While CNNs have pushed the state-of-the-art forward, their training process requires a large amount of clean and correctly labelled training data. If a CNN is intended to tolerate facial pose, then we face an important question: should this training data be diverse in its pose distribution, or should face images be normalized to a single pose in a pre-processing step? To address this question, we evaluate a number of popular facial landmarking and pose correction algorithms to understand their effect on facial recognition performance. Additionally, we introduce a new, automatic, single-image frontalization scheme that exceeds the performance of current algorithms. CNNs trained using sets of different pre-processing methods are used to extract features from the Point and Shoot Challenge (PaSC) and CMU Multi-PIE datasets. We assert that the subsequent verification and recognition performance serves to quantify the effectiveness of each pose correction scheme.

Motivation & Objective

• To investigate whether pre-processing face images to a canonical frontal pose improves deep learning-based face recognition performance.
• To evaluate existing facial landmarking and pose correction algorithms in the context of recognition accuracy.
• To develop and validate a new, automatic single-image frontalization method that outperforms current state-of-the-art techniques.
• To quantify the effectiveness of different pre-processing strategies through feature extraction and verification performance on benchmark datasets.

Proposed method

• The authors apply multiple facial landmarking and pose correction algorithms to normalize face images to a frontal pose prior to CNN feature extraction.
• A novel single-image frontalization method is introduced, leveraging deep learning to synthesize frontal views from single, unconstrained images.
• CNNs are trained on datasets pre-processed with various frontalization techniques, including the proposed method, to extract discriminative features.
• Performance is evaluated using the Point and Shoot Challenge (PaSC) and CMU Multi-PIE datasets, focusing on verification and recognition accuracy.
• The effectiveness of each pre-processing method is quantified by comparing recognition performance across different pose conditions.

Experimental results

Research questions

• RQ1 Does pre-processing face images to a frontal pose improve recognition performance in deep learning models?
• RQ2 How do existing facial landmarking and pose correction algorithms compare in their impact on recognition accuracy?
• RQ3 Can a new single-image frontalization method outperform existing state-of-the-art techniques?
• RQ4 What is the relative contribution of pose normalization versus diverse pose training data in CNN-based face recognition?

Key findings

• The proposed single-image frontalization method achieves superior performance compared to existing algorithms in terms of recognition accuracy.
• Pre-processing with frontalization significantly enhances recognition performance on unconstrained datasets like PaSC and Multi-PIE.
• Pose normalization through pre-processing leads to more robust feature representations, especially under large pose variations.
• The study demonstrates that training with pre-processed frontal images yields better verification performance than training with raw, unconstrained poses.