[Paper Review] Human Pose Estimation from Depth Images via Inference Embedded Multi-task Learning

Keze Wang, Shengfu Zhai+3 more|arXiv (Cornell University)|Aug 13, 2016

Human Pose and Action Recognition39 references4 citations

TL;DR

This paper proposes an inference-embedded multi-task deep learning framework for human pose estimation from depth images, combining a fully convolutional network (FCN) for heat map prediction with a novel MatchNet that embeds dynamic programming inference to jointly optimize appearance and 3D kinematic compatibility. The method achieves state-of-the-art accuracy on challenging depth data, outperforming existing SDKs and methods, and is 190x faster than comparable inference-heavy approaches.

ABSTRACT

Human pose estimation (i.e., locating the body parts / joints of a person) is a fundamental problem in human-computer interaction and multimedia applications. Significant progress has been made based on the development of depth sensors, i.e., accessible human pose prediction from still depth images [32]. However, most of the existing approaches to this problem involve several components/models that are independently designed and optimized, leading to suboptimal performances. In this paper, we propose a novel inference-embedded multi-task learning framework for predicting human pose from still depth images, which is implemented with a deep architecture of neural networks. Specifically, we handle two cascaded tasks: i) generating the heat (confidence) maps of body parts via a fully convolutional network (FCN); ii) seeking the optimal configuration of body parts based on the detected body part proposals via an inference built-in MatchNet [10], which measures the appearance and geometric kinematic compatibility of body parts and embodies the dynamic programming inference as an extra network layer. These two tasks are jointly optimized. Our extensive experiments show that the proposed deep model significantly improves the accuracy of human pose estimation over other several state-of-the-art methods or SDKs. We also release a large-scale dataset for comparison, which includes 100K depth images under challenging scenarios.

Motivation & Objective

• Address the limitations of existing human pose estimation methods on depth images, particularly in handling unusual poses and self-occlusions.
• Overcome suboptimal performance from independently optimized components in traditional approaches.
• Develop a unified deep learning framework that jointly optimizes feature learning and structural constraints.
• Improve robustness and accuracy in complex scenarios such as occlusions and non-standard body poses.
• Enable efficient inference by embedding dynamic programming into the neural network architecture.

Proposed method

• Use a fully convolutional network (FCN) to generate heat maps representing confidence scores for all body parts in a depth image.
• Apply a novel inference-built-in MatchNet that integrates appearance matching and 3D kinematic constraints as part of the network layers.
• Embed dynamic programming inference as a differentiable layer within the MatchNet to find the optimal body part configuration.
• Jointly optimize both the FCN and MatchNet components end-to-end using a multi-task learning objective.
• Leverage geometric and appearance compatibility between part proposals and templates to refine final pose estimation.
• Introduce a 3D kinematic constraint model that enforces plausible joint relationships based on human body structure.

Experimental results

Research questions

• RQ1 Can a deep learning framework jointly optimize feature extraction and structural reasoning for human pose estimation from depth images?
• RQ2 How does embedding inference (dynamic programming) directly into the neural network architecture improve pose estimation accuracy and efficiency?
• RQ3 To what extent does incorporating 3D kinematic constraints enhance performance on challenging poses and occlusions?
• RQ4 How does the proposed method compare to state-of-the-art methods and commercial SDKs in accuracy and inference speed?
• RQ5 What is the contribution of appearance matching versus geometric constraints in the final pose estimation?

Key findings

• The proposed method achieves state-of-the-art performance on the K2HGD dataset, significantly outperforming the latest Microsoft Kinect SDK and other SOTA methods.
• The method is 190x faster than Chen et al.'s approach and 5x faster than Tompson et al.'s method, despite incorporating an inference step.
• The inference-embedded MatchNet improves accuracy over FCN-only and FCN+standard-MatchNet baselines, especially for key joints like elbow, shoulder, knee, and foot.
• The 3D kinematic constraint component contributes significantly to performance, demonstrating the importance of structural priors in pose estimation.
• The method shows robustness in handling self-occlusions and unusual poses, where commercial SDKs often fail.
• The ablation study confirms that both appearance matching and geometric compatibility are essential for optimal performance.