[Paper Review] A Vector-based Representation to Enhance Head Pose Estimation

Zhiwen Cao, Zongcheng Chu+2 more|arXiv (Cornell University)|Oct 14, 2020

Face recognition and analysis41 references4 citations

TL;DR

This paper proposes a vector-based representation using three orthonormal vectors (left, down, front) to represent head pose, overcoming discontinuities in Euler angles and quaternions. It introduces a new loss metric, Mean Absolute Error of Vectors (MAEV), and a constrained neural network (TriNet) that predicts orthonormal vectors, achieving state-of-the-art performance on AFLW2000 and BIWI datasets with significantly reduced errors on profile views.

ABSTRACT

This paper proposes to use the three vectors in a rotation matrix as the representation in head pose estimation and develops a new neural network based on the characteristic of such representation. We address two potential issues existed in current head pose estimation works: 1. Public datasets for head pose estimation use either Euler angles or quaternions to annotate data samples. However, both of these annotations have the issue of discontinuity and thus could result in some performance issues in neural network training. 2. Most research works report Mean Absolute Error (MAE) of Euler angles as the measurement of performance. We show that MAE may not reflect the actual behavior especially for the cases of profile views. To solve these two problems, we propose a new annotation method which uses three vectors to describe head poses and a new measurement Mean Absolute Error of Vectors (MAEV) to assess the performance. We also train a new neural network to predict the three vectors with the constraints of orthogonality. Our proposed method achieves state-of-the-art results on both AFLW2000 and BIWI datasets. Experiments show our vector-based annotation method can effectively reduce prediction errors for large pose angles.

Motivation & Objective

• To address the discontinuity issues in Euler angle and quaternion annotations used in head pose estimation.
• To propose a new evaluation metric, MAEV (Mean Absolute Error of Vectors), that better reflects actual pose estimation performance, especially for large pose angles.
• To develop a deep neural network (TriNet) that predicts three orthonormal vectors representing head pose with orthogonal constraints.
• To improve performance on profile view images, where traditional Euler angle-based methods fail due to gimbal lock and label discontinuity.

Proposed method

• Represent head pose using three orthonormal vectors: left (red), down (green), and front (blue), derived from the rotation matrix.
• Train a CNN-based network (TriNet) to predict these three vectors directly, enforcing orthogonality via a differentiable loss constraint.
• Introduce a new loss function based on the angle difference between predicted and ground-truth vectors, forming the MAEV metric.
• Apply uniform sampling in the capsule network output layer to improve feature representation and prediction accuracy.
• Use 300W-LP as the training dataset and evaluate on AFLW2000 and BIWI datasets with both MAE and MAEV metrics.
• Constrain the predicted vectors to remain orthonormal during training using a differentiable orthogonality regularization loss.

Experimental results

Research questions

• RQ1 Can a vector-based representation of head pose reduce discontinuity issues present in Euler angles and quaternions?
• RQ2 Does the proposed MAEV metric provide a more accurate and reliable evaluation of head pose estimation performance than traditional MAE of Euler angles?
• RQ3 Can a deep neural network trained to predict three orthonormal vectors achieve better performance on large pose angles, especially profile views?
• RQ4 How does the inclusion of orthogonality constraints and vector-based loss improve generalization and robustness in head pose estimation?

Key findings

• The proposed vector-based annotation method results in consistent representations for similar poses, such as profile views, unlike Euler angles which show large label discrepancies.
• The MAEV metric provides a more accurate performance measure, with a prediction error of only 2.249° for profile images, compared to a misleading 44.2° MAE in Euler angles.
• On the BIWI dataset, TriNet achieves the best performance with a MAE of 4.23° and MAEV of 2.81°, outperforming all baseline methods.
• Error analysis shows that MAEV increases much more slowly than MAE with increasing pose angles, confirming MAE's inadequacy for large pose estimation.
• Ablation studies confirm that combining feature mapping, capsule module, and orthogonality loss yields the best performance, with non-uniform sampling in the capsule layer further improving results.
• The method significantly reduces prediction errors for profile views, where FSA-Net and Hopenet produce arbitrary results, while TriNet maintains high accuracy.