[Paper Review] Dense 3D Regression for Hand Pose Estimation

Chengde Wan, Thomas Probst+2 more|arXiv (Cornell University)|Nov 24, 2017

Human Pose and Action Recognition38 references3 citations

TL;DR

This paper proposes a dense 3D regression method for single-depth-frame hand pose estimation by modeling 2D/3D joint heat maps and unit 3D directional vector fields via a multi-task CNN cascade. The approach uses a mean shift variant to aggregate pixel-wise votes while enforcing consistency between 2D and 3D predictions, achieving state-of-the-art accuracy on MSRA, NYU, and ICVL datasets with average errors of 7.2mm, 10.2mm, and 7.3mm respectively.

ABSTRACT

We present a simple and effective method for 3D hand pose estimation from a single depth frame. As opposed to previous state-of-the-art methods based on holistic 3D regression, our method works on dense pixel-wise estimation. This is achieved by careful design choices in pose parameterization, which leverages both 2D and 3D properties of depth map. Specifically, we decompose the pose parameters into a set of per-pixel estimations, i.e., 2D heat maps, 3D heat maps and unit 3D directional vector fields. The 2D/3D joint heat maps and 3D joint offsets are estimated via multi-task network cascades, which is trained end-to-end. The pixel-wise estimations can be directly translated into a vote casting scheme. A variant of mean shift is then used to aggregate local votes while enforcing consensus between the the estimated 3D pose and the pixel-wise 2D and 3D estimations by design. Our method is efficient and highly accurate. On MSRA and NYU hand dataset, our method outperforms all previous state-of-the-art approaches by a large margin. On the ICVL hand dataset, our method achieves similar accuracy compared to the currently proposed nearly saturated result and outperforms various other proposed methods. Code is available $\href{"https://github.com/melonwan/denseReg"}{ ext{online}}$.

Motivation & Objective

• To address the limitations of holistic 3D regression and 2D detection in single-depth-frame hand pose estimation.
• To improve robustness to hand pose variations, translational variance, and self-occlusion by combining 2D and 3D geometric properties of depth maps.
• To enable end-to-end learning of joint detection and 3D regression through a unified, dense pixel-wise estimation framework.
• To reduce reliance on discrete volumetric representations and avoid the computational burden and ambiguity of 3D CNNs.

Proposed method

• Reparameterize 3D joint positions as per-pixel 3D offsets instead of absolute coordinates to ensure translational invariance.
• Predict 2D joint heat maps, 3D joint heat maps (based on 3D distance), and unit 3D directional vector fields for each pixel.
• Use a multi-task, end-to-end trainable CNN cascade to regress all three components simultaneously.
• Apply a non-parametric mean shift variant to aggregate local votes from heat maps and vector fields, enforcing consistency between 2D and 3D predictions.
• Integrate 2D detection and 3D regression via a vote-casting scheme that enforces geometric consistency in the final pose estimate.
• Leverage the 2.5D nature of depth maps by combining 2D CNNs for local pattern learning with 3D geometric reasoning for final pose regression.

Experimental results

Research questions

• RQ1 Can dense pixel-wise estimation of 2D and 3D joint properties improve 3D hand pose estimation accuracy compared to holistic regression?
• RQ2 How can 2D and 3D geometric priors from depth maps be effectively fused in a deep learning framework without using 3D CNNs?
• RQ3 Can a vote-casting aggregation scheme based on heat maps and directional fields improve robustness to self-occlusion and pose variation?
• RQ4 To what extent can offset-based regression with unit vector fields stabilize training and improve generalization across diverse hand gestures?
• RQ5 Does combining 2D detection and 3D regression via a consistent voting mechanism outperform cascaded or shared-feature approaches?

Key findings

• On the MSRA dataset, the proposed method achieves an average 3D error of 7.2mm, outperforming all prior state-of-the-art methods.
• On the NYU dataset, the method achieves a 10.2mm average 3D error, significantly improving over previous SOTA results.
• On the ICVL dataset, the method attains a 7.3mm average 3D error, matching the nearly saturated performance of Chen et al. [5] while outperforming other methods.
• Over 81% of frames on MSRA have joint errors below 20mm, a substantial improvement over the 60% achieved by the prior SOTA method.
• The method achieves high success rates on challenging poses, with 91% of frames having errors below 30mm on MSRA, compared to 81% for the previous best method.
• Qualitative results show consistent performance across diverse hand gestures and self-occlusion scenarios, with only minor failures in extreme cases.