[Paper Review] Facial Pose Estimation by Deep Learning from Label Distributions

Zhaoxiang Liu, Zezhou Chen+3 more|arXiv (Cornell University)|Apr 30, 2019

Face recognition and analysis54 references4 citations

TL;DR

This paper proposes a deep learning approach for facial pose estimation by modeling pose regression as label distribution learning, using a Gaussian label distribution loss to improve generalization on imbalanced datasets. By training a CNN on AFLW and 300W-LP with this loss, the method achieves state-of-the-art performance, reducing mean absolute error (MAE) by up to 13.8% on benchmarks like AFLW2000 and BIWI.

ABSTRACT

Facial pose estimation has gained a lot of attentions in many practical applications, such as human-robot interaction, gaze estimation and driver monitoring. Meanwhile, end-to-end deep learning-based facial pose estimation is becoming more and more popular. However, facial pose estimation suffers from a key challenge: the lack of sufficient training data for many poses, especially for large poses. Inspired by the observation that the faces under close poses look similar, we reformulate the facial pose estimation as a label distribution learning problem, considering each face image as an example associated with a Gaussian label distribution rather than a single label, and construct a convolutional neural network which is trained with a multi-loss function on AFLW dataset and 300W-LP dataset to predict the facial poses directly from color image. Extensive experiments are conducted on several popular benchmarks, including AFLW2000, BIWI, AFLW and AFW, where our approach shows a significant advantage over other state-of-the-art methods.

Motivation & Objective

• Address the challenge of insufficient training data for large facial poses in existing datasets like AFLW and 300W-LP.
• Overcome the limitations of standard softmax cross-entropy loss, which fails to exploit similarity between adjacent poses.
• Improve pose estimation accuracy by modeling pose labels as continuous distributions rather than discrete single labels.
• Demonstrate that label distribution learning with a Gaussian constraint enhances model robustness and generalization, especially for rare or large poses.

Proposed method

• Reformulate facial pose estimation as a label distribution learning problem, where each face image is associated with a Gaussian-distributed label centered on the true pose.
• Define a novel Gaussian label distribution loss that encourages predictions to be close to the true pose and to smoothly decay for neighboring poses.
• Train a convolutional neural network with a multi-loss function combining the Gaussian label distribution loss and standard regression loss.
• Use data from AFLW and 300W-LP datasets to train the model end-to-end on RGB images for direct regression of yaw, pitch, and roll angles.
• Introduce a hyperparameter α to control the width of the Gaussian distribution, allowing trade-off between localization precision and generalization.
• Apply the trained model to test on multiple benchmarks including AFLW2000, BIWI, AFLW, and AFW, evaluating performance using yaw, pitch, roll, and MAE metrics.

Experimental results

Research questions

• RQ1 Can modeling facial pose as a label distribution rather than a single label improve performance on imbalanced datasets?
• RQ2 Does incorporating a Gaussian-based similarity constraint between adjacent poses reduce error in pose estimation, especially for underrepresented large poses?
• RQ3 How does the proposed label distribution loss compare to standard softmax cross-entropy and regression losses in terms of robustness and accuracy?
• RQ4 To what extent does the method generalize across diverse benchmarks such as AFLW2000, BIWI, and AFW?
• RQ5 Why does the performance gain diminish for roll estimation compared to yaw and pitch, and can this be attributed to data imbalance?

Key findings

• The proposed method achieves state-of-the-art performance on AFLW2000, reducing MAE by 13.8% compared to the best baseline (Hopenet).
• On the BIWI benchmark, the method reduces MAE by 19.4% for pitch and 20.8% for yaw compared to Hopenet, with a 13.8% MAE reduction overall.
• On the AFLW benchmark, the best model (α=0.01) reduces yaw error by 4.2%, pitch by 9.85%, and MAE by 5.71% compared to Hopenet.
• On the AFW benchmark, the method achieves over 99% accuracy, outperforming all baselines, including Hopenet and KEPLER.
• The model with α=0.01 achieves the best balance between precision and generalization, showing consistent improvements across all metrics and datasets.
• The performance gap for roll estimation is larger than for yaw and pitch, primarily due to extreme data imbalance in the roll domain, where 65.57% of AFLW and 67.65% of BIWI examples lie within ±10°.