[Paper Review] 3D Hand Shape and Pose Estimation from a Single RGB Image

Liuhao Ge, Zhou Ren+5 more|arXiv (Cornell University)|Mar 3, 2019

Human Pose and Action Recognition58 references33 citations

TL;DR

This paper proposes a Graph CNN-based approach to estimate full 3D hand mesh and 3D hand pose from a single RGB image, trained on a large synthetic dataset and fine-tuned with weak supervision from depth maps on real data.

ABSTRACT

This work addresses a novel and challenging problem of estimating the full 3D hand shape and pose from a single RGB image. Most current methods in 3D hand analysis from monocular RGB images only focus on estimating the 3D locations of hand keypoints, which cannot fully express the 3D shape of hand. In contrast, we propose a Graph Convolutional Neural Network (Graph CNN) based method to reconstruct a full 3D mesh of hand surface that contains richer information of both 3D hand shape and pose. To train networks with full supervision, we create a large-scale synthetic dataset containing both ground truth 3D meshes and 3D poses. When fine-tuning the networks on real-world datasets without 3D ground truth, we propose a weakly-supervised approach by leveraging the depth map as a weak supervision in training. Through extensive evaluations on our proposed new datasets and two public datasets, we show that our proposed method can produce accurate and reasonable 3D hand mesh, and can achieve superior 3D hand pose estimation accuracy when compared with state-of-the-art methods.

Motivation & Objective

• Motivate the need for full 3D hand mesh estimation from monocular RGB images for VR/AR applications.
• Develop an end-to-end framework that regresses dense 3D hand mesh vertices and 3D hand joints from a single image.
• Leverage a large synthetic dataset with ground-truth meshes and poses for supervision.
• Introduce a weakly-supervised fine-tuning strategy using depth maps to bridge the synthetic-real gap.
• Publish a real-time capable method suitable for practical deployment.

Proposed method

• Use a two-stacked hourglass network to extract 2D heatmaps and image features from an input hand-centered RGB image.
• Encode features into a latent vector and predict 3D hand mesh vertices via a Graph CNN with a hierarchical, upsampling-based graph architecture.
• Predefine a fixed hand mesh topology and perform graph convolutions to generate 1280 3D mesh vertices from the latent features.
• Linearly regress 3D hand joints from the reconstructed mesh using a simplified Graph CNN regressor.
• Train on a large synthetic dataset with heatmap, mesh, and pose losses; fine-tune on real data with weak supervision by rendering meshes to depth maps and using a pseudo-ground-truth mesh as additional guidance.

Experimental results

Research questions

• RQ1 Can a Graph CNN-based model directly generate full 3D hand mesh coordinates from a single RGB image?
• RQ2 Does incorporating full 3D hand mesh improve 3D hand pose estimation accuracy over joint-only approaches?
• RQ3 How can synthetic data be leveraged to train for real-world RGB hand mesh estimation, and how can depth-based weak supervision bridge the domain gap?
• RQ4 What is the effect of pseudo-ground-truth mesh supervision on fine-tuning with depth maps?
• RQ5 Is the proposed method capable of real-time performance on common GPUs?

Key findings

• The method achieves accurate and reasonable 3D hand meshes and superior 3D hand pose estimation compared to state-of-the-art methods on public datasets.
• Graph CNN-based mesh generation better represents hand shape variations and local details than direct LBS or MANO-based approaches.
• Weakly-supervised fine-tuning with depth-map loss and pseudo-ground-truth mesh loss improves pose estimation when 3D supervision is unavailable on real data.
• The approach runs in real-time on GPU at over 50 fps, with 19.9 ms average runtime for full mesh+pose output.
• A large synthetic dataset (375,000 hand RGB images) with mesh and pose annotations supports effective end-to-end training.
• On RHD and STB datasets, the method outperforms several state-of-the-art 3D hand pose methods, particularly when 3D pose supervision is limited.