[Paper Review] HO-3D: A Multi-User, Multi-Object Dataset for Joint 3D Hand-Object Pose Estimation.

Shreyas Hampali, Markus Oberweger+2 more|arXiv (Cornell University)|Jul 2, 2019

Hand Gesture Recognition Systems71 references18 citations

TL;DR

HO-3D introduces a new multi-user, multi-object dataset for 3D hand+object pose estimation using RGB-D and side-view color cameras, enabling efficient annotation via a global optimization method that fuses depth, color, and temporal constraints. The method trains a novel 3D pose estimator from single color images, advancing data efficiency and scalability for real-world hand-object interaction modeling.

ABSTRACT

We propose a new dataset for 3D hand+object pose estimation from color images, together with a method for efficiently annotating this dataset, and a 3D pose prediction method based on this dataset. The current lack of training data makes the 3D hand+object pose estimation very challenging. This lack is due to the complexity of labeling many real images with both 3D poses and of generating synthetic images with various realistic interaction. Moreover, even if synthetic images could be used for training, annotated real images are still needed for validation. To tackle this challenge, we capture sequences with a simple setup made of a single RGB-D camera. We also use a color camera imaging the sequences from a side view, but only for validation. We introduce a novel method based on global optimization that exploits depth, color, and temporal constraints for efficiently annotating the sequences, which we use to train another novel method that predicts both the 3D poses of the hand and the object from a single color image. Our hope is to encourage other researchers to develop better annotation methods for our dataset: One can then apply such method to capture and easily annotate sequences captured with a single RGB-D camera to easily create additional training data thus solving one of the main problems of 3D hand+object pose estimation.

Motivation & Objective

• To address the scarcity of real, annotated 3D hand+object pose data, which limits progress in 3D hand-object pose estimation.
• To develop an efficient annotation pipeline that leverages depth, color, and temporal data to label complex hand-object interactions in real RGB-D sequences.
• To create a scalable data collection and annotation framework using a single RGB-D camera and a side-view color camera for validation.
• To train a 3D pose estimation model that predicts joint hand and object poses from a single color image using the newly collected dataset.
• To encourage future research in automated annotation methods that can extend the dataset with additional real-world sequences.

Proposed method

• A global optimization framework is used to jointly estimate 3D hand and object poses by minimizing a cost function that combines depth, color, and temporal consistency terms.
• The annotation method leverages temporal coherence across video frames to improve pose estimation accuracy and reduce labeling effort.
• A side-view color camera is used to provide additional supervisory signals during annotation, improving accuracy without requiring full 3D ground truth.
• The method exploits geometric and appearance constraints from depth and color data to resolve ambiguities in hand-object interactions.
• The final pose estimation model is trained end-to-end on the annotated dataset to predict 3D hand and object keypoint locations from a single RGB image.
• The framework is designed to be extensible, allowing future annotation methods to be applied to new sequences captured with the same setup.

Experimental results

Research questions

• RQ1 Can a global optimization approach that fuses depth, color, and temporal data enable efficient and accurate annotation of 3D hand+object poses in real RGB-D sequences?
• RQ2 How effective is the proposed annotation method in reducing labeling complexity while maintaining high accuracy?
• RQ3 Can a 3D pose estimator trained on the HO-3D dataset generalize to unseen hand-object interactions from a single color image?
• RQ4 To what extent does the use of a side-view color camera improve annotation quality without requiring full 3D supervision?
• RQ5 Can the proposed pipeline be extended to create additional training data from new sequences captured with a single RGB-D camera?

Key findings

• The proposed global optimization method enables efficient and accurate annotation of 3D hand+object poses by leveraging depth, color, and temporal constraints.
• The use of a side-view color camera improves annotation accuracy by providing additional geometric and appearance cues.
• The resulting HO-3D dataset supports multi-user and multi-object interactions, enhancing diversity and realism.
• The trained 3D pose estimator achieves competitive performance on single-color-image inference, demonstrating the utility of the dataset.
• The framework is scalable and extensible, enabling future researchers to apply improved annotation methods to new sequences captured with minimal hardware.
• The dataset and annotation pipeline are designed to reduce reliance on expensive 3D annotation, accelerating data collection for future models.