[Paper Review] Two-hand Global 3D Pose Estimation Using Monocular RGB

Fanqing Lin, Connor Wilhelm+1 more|arXiv (Cornell University)|Jun 1, 2020

Human Pose and Action Recognition30 references4 citations

TL;DR

This paper presents the first system for estimating 3D global joint locations of both hands from a single monocular RGB image, using a multi-stage CNN pipeline with a novel projection algorithm to reconstruct absolute 3D poses without depth sensors. It achieves state-of-the-art performance on 3D canonical pose estimation and introduces a new benchmark for global 3D hand pose estimation using a large-scale synthetic egocentric dataset.

ABSTRACT

We tackle the challenging task of estimating global 3D joint locations for both hands via only monocular RGB input images. We propose a novel multi-stage convolutional neural network based pipeline that accurately segments and locates the hands despite occlusion between two hands and complex background noise and estimates the 2D and 3D canonical joint locations without any depth information. Global joint locations with respect to the camera origin are computed using the hand pose estimations and the actual length of the key bone with a novel projection algorithm. To train the CNNs for this new task, we introduce a large-scale synthetic 3D hand pose dataset. We demonstrate that our system outperforms previous works on 3D canonical hand pose estimation benchmark datasets with RGB-only information. Additionally, we present the first work that achieves accurate global 3D hand tracking on both hands using RGB-only inputs and provide extensive quantitative and qualitative evaluation.

Motivation & Objective

• To enable accurate 3D global hand pose estimation for both hands using only monocular RGB input, overcoming challenges from occlusion, background noise, and lack of depth.
• To address the lack of large-scale, real-world annotated data for two-hand 3D pose estimation by creating a high-quality synthetic egocentric dataset.
• To develop a novel viewpoint-invariant projection algorithm that reconstructs absolute 3D joint locations from 2D and 3D canonical poses.
• To establish a new evaluation protocol and benchmark for 3D global hand pose estimation using RGB-only inputs.

Proposed method

• A multi-stage CNN pipeline: (1) hand segmentation and detection, (2) 2D hand pose estimation, (3) 3D canonical pose estimation, and (4) 3D global pose estimation via projection.
• A novel global projection algorithm that computes absolute 3D joint locations using estimated 3D canonical poses, 2D joint locations, and known bone lengths.
• Training with a synthetic egocentric RGB-D dataset that includes both static and dynamic hand poses, simulating real-world viewing conditions.
• Use of a novel loss function, $\mathcal{L}_{proj}$, that improves global pose estimation accuracy despite slight degradation in canonical pose performance.
• Incorporation of hand segmentation and batch normalization layers to enhance 2D and 3D pose estimation accuracy.
• Spherical coordinate-based evaluation for global pose estimation to assess directional and distance accuracy of the root joint (mMCP).

Experimental results

Research questions

• RQ1 Can 3D global hand pose estimation for both hands be accurately achieved using only monocular RGB input, without depth or multi-view data?
• RQ2 How can absolute 3D joint locations be reconstructed from 2D and 3D canonical poses when depth is unavailable?
• RQ3 To what extent does a synthetic egocentric dataset with realistic texture and viewpoint variation enable generalization to real-world RGB data?
• RQ4 What is the impact of different loss functions and network components on the accuracy of 2D, 3D canonical, and global 3D pose estimation?
• RQ5 How does the proposed method compare to state-of-the-art approaches on both canonical and global 3D pose estimation benchmarks?

Key findings

• The proposed method achieves a spherical PCK of approximately 0.90 at 3° angular threshold and 7 cm radius threshold on the $Ego3D_{s}$-test and $Ego3D_{d}$-test datasets for global 3D pose estimation.
• The method outperforms state-of-the-art single-hand 3D canonical pose estimation methods that use additional depth or multi-view data, despite relying only on RGB inputs.
• The use of $\mathcal{L}_{proj}$ loss improved global pose estimation accuracy, increasing spherical PCK despite a slight drop in canonical pose AUC and EPE.
• On the STB dataset, the method achieved a spherical PCK of 0.90 at 3° and 7 cm thresholds without access to ground truth root joint locations, demonstrating generalization to real-world data.
• On the RHP dataset, the method achieved an AUC of 0.960 for azimuth ($\theta$), 0.958 for elevation ($\phi$), and 0.690 for radius, indicating strong directional accuracy but room for improvement in depth estimation.
• Ablation studies confirmed that hand segmentation and batch normalization significantly improved 2D and 3D pose estimation performance, and $\mathcal{L}_{bone}$ reduced average bone length error from 3.7mm to 1.8mm.