[Paper Review] Face Alignment Assisted by Head Pose Estimation

Heng Yang, Wenxuan Mou+4 more|arXiv (Cornell University)|Jul 11, 2015

Face recognition and analysis27 references18 citations

TL;DR

This paper proposes a supervised initialisation scheme for cascaded face alignment using explicit head pose estimation via a deep ConvNet. By projecting a 3D face shape or retrieving nearest neighbour shapes from training data based on estimated head pose, the method improves alignment robustness under large pose variations, reducing failure cases by 50% on the 300W dataset while achieving competitive performance.

ABSTRACT

In this paper we propose a supervised initialization scheme for cascaded face alignment based on explicit head pose estimation. We first investigate the failure cases of most state of the art face alignment approaches and observe that these failures often share one common global property, i.e. the head pose variation is usually large. Inspired by this, we propose a deep convolutional network model for reliable and accurate head pose estimation. Instead of using a mean face shape, or randomly selected shapes for cascaded face alignment initialisation, we propose two schemes for generating initialisation: the first one relies on projecting a mean 3D face shape (represented by 3D facial landmarks) onto 2D image under the estimated head pose; the second one searches nearest neighbour shapes from the training set according to head pose distance. By doing so, the initialisation gets closer to the actual shape, which enhances the possibility of convergence and in turn improves the face alignment performance. We demonstrate the proposed method on the benchmark 300W dataset and show very competitive performance in both head pose estimation and face alignment.

Motivation & Objective

• To address the high failure rate of state-of-the-art face alignment methods under large head pose variations.
• To improve cascaded face alignment performance by replacing mean or random initialisation with pose-aware initialisation.
• To develop a reliable deep ConvNet for head pose estimation in unconstrained environments.
• To demonstrate that pose-based initialisation enhances convergence and alignment accuracy, especially for challenging poses.
• To show that the proposed initialisation scheme is generalizable across cascaded face alignment frameworks.

Proposed method

• A deep convolutional neural network (ConvNet) is trained to estimate head pose (yaw, pitch, roll) directly from 2D face images.
• The estimated head pose is used to project a canonical 3D face shape (with 3D landmarks) onto the 2D image plane within the detected face bounding box.
• An alternative initialisation scheme retrieves the nearest neighbour shape from the training set based on head pose distance in the 3D pose space.
• The proposed initialisation is integrated into the Robust Cascaded Pose Regression (RCPR) framework to evaluate performance.
• Both initialisation schemes are evaluated on the 300W benchmark, comparing failure rates and alignment accuracy.
• The method is tested under varying face detection conditions, including Viola-Jones and tight bounding boxes from HeadHunter, to assess robustness.

Experimental results

Research questions

• RQ1 Can explicit head pose estimation improve the reliability of cascaded face alignment under large pose variations?
• RQ2 Does pose-based initialisation reduce failure cases compared to mean or random initialisation?
• RQ3 How does the performance of pose-based initialisation compare to state-of-the-art face alignment methods across different face detection schemes?
• RQ4 Can the proposed initialisation scheme be generalized to other cascaded face alignment frameworks?
• RQ5 What is the impact of head pose estimation accuracy on the final alignment performance?

Key findings

• The proposed method reduces the number of failure cases (alignment error > 0.1) from 130 to 69 using the 3D shape projection initialisation, representing a 50% reduction.
• The nearest neighbour initialisation scheme also reduces failures from 130 to 72, confirming consistent improvement across both pose-based strategies.
• The head pose estimation model achieves a 4° absolute mean error on unconstrained face images, enabling reliable initialisation.
• The method achieves competitive performance compared to recent state-of-the-art face alignment models, including SDM, IFA, LBF, CFAN, and TCDCN.
• The proposed method shows superior robustness when tested with HeadHunter face detection, maintaining stable performance despite tighter and more variable bounding boxes.
• The entire pipeline runs in 3.8 ms per image (0.3 ms for pose estimation, 3.5 ms for alignment), indicating real-time feasibility.