[Paper Review] RGB-based 3D Hand Pose Estimation via Privileged Learning with Depth Images

Shanxin Yuan, Björn Stenger+1 more|arXiv (Cornell University)|Nov 18, 2018

Human Pose and Action Recognition47 references15 citations

TL;DR

This paper proposes a novel RGB-based 3D hand pose estimation method that leverages depth images as privileged information during training to improve performance on RGB-only test data. By pre-training a depth-based network on large-scale depth datasets and transferring its mid-level features to an RGB network via paired RGB-D data, while also using depth-derived hand masks to suppress background responses, the method achieves state-of-the-art performance on 3D and 2D hand pose estimation benchmarks with only RGB input at inference.

ABSTRACT

This paper proposes a method for hand pose estimation from RGB images that uses both external large-scale depth image datasets and paired depth and RGB images as privileged information at training time. We show that providing depth information during training significantly improves performance of pose estimation from RGB images during testing. We explore different ways of using this privileged information: (1) using depth data to initially train a depth-based network, (2) using the features from the depth-based network of the paired depth images to constrain mid-level RGB network weights, and (3) using the foreground mask, obtained from the depth data, to suppress the responses from the background area. By using paired RGB and depth images, we are able to supervise the RGB-based network to learn middle layer features that mimic that of the corresponding depth-based network, which is trained on large-scale, accurately annotated depth data. During testing, when only an RGB image is available, our method produces accurate 3D hand pose predictions. Our method is also tested on 2D hand pose estimation. Experiments on three public datasets show that the method outperforms the state-of-the-art methods for hand pose estimation using RGB image input.

Motivation & Objective

• Address the performance gap between 3D hand pose estimation from RGB and depth images by leveraging abundant, fully annotated depth datasets.
• Overcome the limitation of small-scale, real-world RGB datasets with accurate annotations for 3D hand pose estimation.
• Introduce a new training paradigm that uses depth data as privileged information during model training to improve generalization on RGB-only test data.
• Demonstrate that mid-level feature transfer from a depth-based network to an RGB-based network significantly enhances pose estimation accuracy.
• Explore the impact of foreground hand masks derived from depth data in suppressing background interference during feature learning.

Proposed method

• Pre-train a depth-based 3D hand pose network on large-scale, fully annotated depth datasets (e.g., BigHand2.2M) to learn robust mid-level features.
• Use paired RGB and depth images during a second training stage to align the mid-level activations of the RGB network with those of the pre-trained depth network via a feature regression loss.
• Introduce a foreground hand mask loss that suppresses background responses in the RGB network by leveraging depth-based hand segmentation.
• Freeze the depth-based network after pre-training and fine-tune the RGB network using a joint loss combining 3D pose loss and intermediate feature loss.
• Embed the privileged learning strategy into existing CNN architectures (e.g., CPM) for both 2D and 3D hand pose estimation without architectural overhaul.
• Apply the method to both 3D and 2D hand pose estimation tasks, demonstrating transferability across different network designs and datasets.

Experimental results

Research questions

• RQ1 Can depth images, used as privileged information during training, significantly improve 3D hand pose estimation accuracy from single RGB images at test time?
• RQ2 Does transferring mid-level features from a depth-based network to an RGB-based network lead to better generalization than training the RGB network from scratch?
• RQ3 To what extent does using depth-derived hand masks to suppress background activations improve feature learning in RGB-based pose estimation?
• RQ4 How does the proposed privileged learning strategy compare to state-of-the-art RGB-only methods in terms of performance on public benchmarks?
• RQ5 Can the training strategy be effectively transferred to different CNN architectures and 2D hand pose estimation tasks?

Key findings

• The proposed method outperforms state-of-the-art RGB-based 3D hand pose estimation methods on the Stereo, RHD, and Dexter-Object datasets, achieving a median 2D PCK of 11.809 on the Dexter-Object dataset with only RHD training data.
• On the RHD dataset, the method reduced the median 2D end-point error (EPE) to 2.642, compared to 3.708 for the baseline RGB-only model, demonstrating significant improvement from privileged learning.
• The joint loss with mid-level feature regularization converged by around 30,000 iterations, indicating stable optimization during the privileged information fine-tuning stage.
• Using depth-derived hand masks during training reduced background interference, improving feature quality and leading to more accurate keypoint localization in qualitative results.
• The method achieved a 2.087 median EPE on the RHD dataset using only depth-based supervision, and the RGB model with privileged learning reached 2.642, showing that RGB networks can closely mimic depth-based performance when guided by privileged depth data.
• Even with less training data (RHD only), the method outperformed a SOTA model (Z&B_Joint) trained on both RHD and Stereo datasets, highlighting the data efficiency of the privileged learning approach.