[Paper Review] Face Frontalization Based on Robustly Fitting a Deformable Shape Model to 3D Landmarks

Zhiqi Kang, Mostafa Sadeghi+1 more|arXiv (Cornell University)|Oct 26, 2020

Face recognition and analysis32 references4 citations

TL;DR

This paper proposes a robust 3D point-set alignment method for face frontalization by simultaneously estimating rigid transformations (scale, rotation, translation) and non-rigid deformations between 3D facial landmarks using a generalized Student’s t-distribution to model inliers and outliers. The method enables pixel-perfect warping via an EM algorithm, achieving state-of-the-art performance in frontalization quality as measured by zero-mean normalized cross-correlation (ZNCC) on a profile-to-frontal face dataset.

ABSTRACT

Face frontalization consists of synthesizing a frontally-viewed face from an arbitrarily-viewed one. The main contribution of this paper is a robust face alignment method that enables pixel-to-pixel warping. The method simultaneously estimates the rigid transformation (scale, rotation, and translation) and the non-rigid deformation between two 3D point sets: a set of 3D landmarks extracted from an arbitrary-viewed face, and a set of 3D landmarks parameterized by a frontally-viewed deformable face model. An important merit of the proposed method is its ability to deal both with noise (small perturbations) and with outliers (large errors). We propose to model inliers and outliers with the generalized Student's t-probability distribution function, a heavy-tailed distribution that is immune to non-Gaussian errors in the data. We describe in detail the associated expectation-maximization (EM) algorithm that alternates between the estimation of (i) the rigid parameters, (ii) the deformation parameters, and (iii) the Student-t distribution parameters. We also propose to use the zero-mean normalized cross-correlation, between a frontalized face and the corresponding ground-truth frontally-viewed face, to evaluate the performance of frontalization. To this end, we use a dataset that contains pairs of profile-viewed and frontally-viewed faces. This evaluation, based on direct image-to-image comparison, stands in contrast with indirect evaluation, based on analyzing the effect of frontalization on face recognition.

Motivation & Objective

• To develop a robust method for face frontalization that enables pixel-to-pixel warping from arbitrary to frontal views.
• To address both noise and outliers in 3D facial landmark alignment using a heavy-tailed statistical model.
• To provide a direct evaluation of frontalization quality via image-level metrics, contrasting with indirect recognition-based evaluation.
• To enable accurate rigid and non-rigid parameter estimation through joint optimization using an expectation-maximization framework.

Proposed method

• Models inliers and outliers using the generalized Student’s t-probability distribution, which is robust to non-Gaussian errors and heavy-tailed data.
• Employs an expectation-maximization (EM) algorithm that iteratively estimates rigid parameters (scale, rotation, translation), deformation parameters (shape coefficients), and t-distribution parameters (degrees of freedom, location, scale).
• Uses a statistical shape model parameterized by a mean shape and principal component analysis (PCA) basis to represent non-rigid facial deformations.
• Applies zero-mean normalized cross-correlation (ZNCC) between frontalized and ground-truth frontal faces as a direct, image-level evaluation metric.
• Performs dense depth map reconstruction via triangulated mesh interpolation from the frontalized 3D shape model to enable pixel-wise warping.
• Integrates a 99% confidence ellipsoid constraint on the shape parameters to regularize the deformation space and improve robustness.

Experimental results

Research questions

• RQ1 Can a robust 3D point-set alignment method effectively handle both noise and outliers in facial landmark data for accurate face frontalization?
• RQ2 How does modeling inliers and outliers with a generalized Student’s t-distribution improve alignment accuracy compared to Gaussian-based methods?
• RQ3 To what extent does the proposed method achieve pixel-perfect warping for frontalization, as measured by direct image-to-image similarity?
• RQ4 Can the joint estimation of rigid pose and non-rigid deformation via EM yield better frontalization results than state-of-the-art methods?

Key findings

• The proposed method achieves higher zero-mean normalized cross-correlation (ZNCC) scores than two state-of-the-art methods on a profile-to-frontal face dataset, indicating superior image-level frontalization quality.
• The use of the generalized Student’s t-distribution significantly improves robustness to outliers and non-Gaussian errors in 3D landmark data.
• The EM-based optimization framework successfully converges to a stable solution that jointly estimates rigid pose, non-rigid deformation, and statistical model parameters.
• The inclusion of a 99% confidence ellipsoid constraint on the shape parameters enhances the stability and realism of the frontalized facial shapes.
• The direct evaluation using ZNCC provides a reliable, image-level benchmark that complements indirect recognition-based evaluations.