[Paper Review] Deep Head Pose Estimation from Depth Data for In-car Automotive Applications

Marco Venturelli, Guido Borghi+2 more|arXiv (Cornell University)|Mar 6, 2017

Face recognition and analysis4 citations

TL;DR

This paper proposes a real-time, end-to-end deep learning approach for head pose estimation using only raw depth data from a single sensor, bypassing the need for facial landmarks or RGB data. The method employs a lightweight CNN to directly regress pitch, roll, and yaw angles, achieving state-of-the-art accuracy (2.8° ± 3.1° pitch, 2.3° ± 2.9° roll, 3.6° ± 4.1° yaw) on the Biwi Kinect Head Pose dataset with inference time under 10 ms per frame.

ABSTRACT

Recently, deep learning approaches have achieved promising results in various fields of computer vision. In this paper, we tackle the problem of head pose estimation through a Convolutional Neural Network (CNN). Differently from other proposals in the literature, the described system is able to work directly and based only on raw depth data. Moreover, the head pose estimation is solved as a regression problem and does not rely on visual facial features like facial landmarks. We tested our system on a well known public dataset, Biwi Kinect Head Pose, showing that our approach achieves state-of-art results and is able to meet real time performance requirements.

Motivation & Objective

• Address the challenge of real-time, accurate head pose estimation in in-car environments using only depth data.
• Overcome limitations of existing methods that rely on RGB data, facial landmarks, or require manual initialization.
• Develop a deep learning framework that efficiently solves the regression problem of head pose angles without compromising speed.
• Enable robust performance under varying lighting conditions and occlusions typical in automotive settings.

Proposed method

• Utilizes a lightweight Convolutional Neural Network (CNN) architecture trained end-to-end on raw depth images to regress head pose angles directly.
• Treats head pose estimation as a regression task, learning a mapping from depth input to three Euler angles (pitch, roll, yaw) without intermediate landmark detection.
• Employs a dynamic cropping strategy based on ground truth head center to focus the network on the face region, improving localization accuracy.
• Trains the network using a regression loss function to minimize angular error between predicted and ground truth pose values.
• Optimizes for real-time performance by using a shallow network design, achieving inference times below 10 ms per frame on a GPU.
• Validates the method on the Biwi Kinect Head Pose Dataset, using standard evaluation protocols for depth-only head pose estimation.

Experimental results

Research questions

• RQ1 Can a deep learning model achieve high-accuracy head pose estimation using only raw depth data, without relying on RGB or facial landmarks?
• RQ2 How does a CNN-based regression approach compare to traditional methods in terms of accuracy and real-time performance on depth data?
• RQ3 To what extent can a lightweight CNN maintain high accuracy while meeting strict real-time constraints in automotive applications?
• RQ4 How robust is the proposed method to visual artifacts and partial occlusions commonly found in in-car depth data?

Key findings

• The proposed method achieves state-of-the-art performance on the Biwi Kinect Head Pose Dataset, with mean angular errors of 2.8° ± 3.1° for pitch, 2.3° ± 2.9° for roll, and 3.6° ± 4.1° for yaw.
• The model processes depth images in under 10 ms per frame on a GPU, satisfying real-time requirements for in-vehicle monitoring systems.
• The method outperforms other state-of-the-art approaches, including those using both RGB and depth data, demonstrating the effectiveness of depth-only learning.
• The network generalizes well to low-quality depth inputs, showing robustness to noise, holes, and partial occlusions common in automotive environments.
• The absence of facial landmark detection simplifies the pipeline and reduces failure points, enhancing reliability in dynamic driving scenarios.
• The approach is extensible to include temporal modeling and fusion with RGB or infrared data in future work to further improve robustness.