[Paper Review] Hand Pose Estimation: A Survey

Bardia Doosti|arXiv (Cornell University)|Mar 3, 2019

Human Pose and Action Recognition58 references33 citations

TL;DR

A comprehensive survey of hand pose estimation methods, comparing depth-based and RGB-based approaches, and detailing major datasets for 2D/3D hand joint estimation. It covers detection vs. regression, 2D/3D outputs, and 22 hand datasets.

ABSTRACT

The success of Deep Convolutional Neural Networks (CNNs) in recent years in almost all the Computer Vision tasks on one hand, and the popularity of low-cost consumer depth cameras on the other, has made Hand Pose Estimation a hot topic in computer vision field. In this report, we will first explain the hand pose estimation problem and will review major approaches solving this problem, especially the two different problems of using depth maps or RGB images. We will survey the most important papers in each field and will discuss the strengths and weaknesses of each. Finally, we will explain the biggest datasets in this field in detail and list 22 datasets with all their properties. To the best of our knowledge this is the most complete list of all the datasets in the hand pose estimation field.

Motivation & Objective

• Explain the hand pose estimation problem and its variations.
• Review depth-based and RGB-based hand pose estimation methods.
• Discuss detection-based versus regression-based approaches in CNNs.
• Summarize major datasets and their properties for hand pose estimation.
• Highlight strengths, weaknesses, and trends in the field.

Proposed method

• Differentiate detection-based heatmap outputs from regression-based joint coordinates.
• Survey depth-based methods including 3D shape estimation and 3D CNNs (e.g., V2V-PoseNet, HandPointNet).
• Survey RGB-based methods including HandSegNet/PoseNet pipelines and multi-view or synthetic data strategies.
• Discuss data augmentation, synthesis, and domain transfer (GANs, CycleGAN) to mitigate annotation limits.
• Describe RGBD hybrids and fusion approaches (FuseNet) for improved accuracy.
• Highlight datasets and annotation schemes (21 joints, 3D coordinates) and their impact on learning.

Experimental results

Research questions

• RQ1 What are the main methodological categories for hand pose estimation, and how do depth-based and RGB-based approaches differ in problem formulation and performance?
• RQ2 How have datasets and annotation practices evolved, and what is their impact on model generalization?
• RQ3 What strategies (e.g., multi-view, synthetic data, domain transfer) mitigate RGB occlusion and data scarcity in hand pose estimation?
• RQ4 What are the relative strengths and weaknesses of detection-based versus regression-based methods in 2D and 3D hand pose estimation?
• RQ5 How do fusion and 3D representations (e.g., TSDF, PointNet) contribute to estimation accuracy?

Key findings

• Depth-based methods evolved toward 3D hand pose estimation with 3D representations and networks like V2V-PoseNet, achieving strong performance on major datasets.
• RGB-based methods require large datasets and often use hand segmentation and multi-stage pipelines to infer 3D pose from 2D cues.
• occlusion handling and data scarcity have driven the use of multi-camera setups, synthetic data, and domain transfer techniques (GANs, CycleGAN).
• Hybrid RGBD approaches (e.g., FuseNet) attempted to fuse depth and RGB streams but may not outperform single-modality methods in some cases.
• GAN-based and synthetic-to-real data pipelines have become common to augment training and reduce annotation burden.
• The survey notes that despite strong dataset-driven results, real-world performance often lags behind simple baselines, underscoring generalization challenges.