[Paper Review] Silhouette-Net: 3D Hand Pose Estimation from Silhouettes

Kuo‐Wei Lee, Shih‐Hung Liu+2 more|arXiv (Cornell University)|Dec 28, 2019

Human Pose and Action Recognition30 references4 citations

TL;DR

Silhouette-Net proposes an end-to-end deep learning framework that estimates 3D hand poses from binary hand silhouettes alone, without requiring depth maps. By learning implicit depth perception through a Depth Perceptive Network (DPN) and a Residual Prediction Network (RPN), the model achieves state-of-the-art performance on the HIM2017 benchmark, with a mean error of 5.60 mm and max-per-joint error of 7.63 mm.

ABSTRACT

3D hand pose estimation has received a lot of attention for its wide range of applications and has made great progress owing to the development of deep learning. Existing approaches mainly consider different input modalities and settings, such as monocular RGB, multi-view RGB, depth, or point cloud, to provide sufficient cues for resolving variations caused by self occlusion and viewpoint change. In contrast, this work aims to address the less-explored idea of using minimal information to estimate 3D hand poses. We present a new architecture that automatically learns a guidance from implicit depth perception and solves the ambiguity of hand pose through end-to-end training. The experimental results show that 3D hand poses can be accurately estimated from solely {\em hand silhouettes} without using depth maps. Extensive evaluations on the {\em 2017 Hands In the Million Challenge} (HIM2017) benchmark dataset further demonstrate that our method achieves comparable or even better performance than recent depth-based approaches and serves as the state-of-the-art of its own kind on estimating 3D hand poses from silhouettes.

Motivation & Objective

• To address the challenge of 3D hand pose estimation using minimal input, specifically binary hand silhouettes, instead of depth maps or RGB images.
• To investigate whether implicit depth perception can replace explicit depth map supervision for accurate 3D hand pose estimation.
• To develop a low-cost, robust system that maintains high accuracy despite self-occlusion and viewpoint variations using only silhouette inputs.
• To demonstrate that learning the 'concept of depth' is more critical than having precise depth values for accurate 3D pose prediction.

Proposed method

• The model uses a multi-view input strategy, processing silhouettes from frontal and side views to improve spatial understanding.
• A Depth Perceptive Network (DPN) is designed with a U-Net and Feature Pyramid Network (FPN)-like architecture to learn hierarchical, multi-scale spatial representations from silhouettes.
• The DPN generates implicit depth perception guidance that constrains and enhances the prediction of 3D joint positions in the Residual Prediction Network (RPN).
• The entire network is trained end-to-end using silhouettes as input, with optional supervision from ground-truth depth maps during training.
• A latent representation of depth perception is learned from multiple views, enabling the model to infer 3D structure without explicit depth data.
• The final prediction is made via a regression head that outputs 3D coordinates of 21 hand joints using the fused features from DPN and RPN.

Experimental results

Research questions

• RQ1 Can 3D hand pose estimation be accurately performed using only binary hand silhouettes, without any depth or RGB input?
• RQ2 Is implicit depth perception sufficient to achieve performance comparable to depth-based methods in 3D hand pose estimation?
• RQ3 How does multi-view silhouette input compare to single-view input in terms of accuracy and robustness?
• RQ4 To what extent does ground-truth depth supervision during training improve performance when the test input is only silhouettes?
• RQ5 Can a model trained on silhouettes generalize well to handle occlusions and complex hand configurations without explicit depth cues?

Key findings

• Silhouette-Net achieves a mean error of 5.60 mm and a max-per-joint error of 7.63 mm on the HIM2017 benchmark, outperforming several state-of-the-art depth-based methods.
• The multi-view version of Silhouette-Net (using frontal and side views) significantly improves accuracy over the single-view baseline, which reports 7.07 mm mean error.
• Even without ground-truth depth supervision during training, the model achieves strong performance (6.75 mm mean error), indicating that depth perception learning is robust and effective.
• The inclusion of ground-truth depth during training further improves performance to 5.60 mm mean error, showing that supervision enhances but is not essential for the final result.
• The method demonstrates that learning depth perception is more critical than having precise depth maps, as the model generalizes well from silhouettes alone.
• The model’s performance surpasses several RGBD-based state-of-the-art approaches, proving that minimal input can yield high accuracy when combined with effective implicit depth learning.