[Paper Review] ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesis

Kailin Li, Lixin Yang+5 more|arXiv (Cornell University)|Sep 12, 2021

Robot Manipulation and Learning47 references4 citations

TL;DR

ArtiBoost proposes an online data augmentation framework that boosts 3D hand-object pose estimation by dynamically exploring and synthesizing diverse hand-object configurations in a Composited hand-object Configuration and Viewpoint (CCV) space. By iteratively sampling hard-to-distinguish triplets and re-weighting them based on model loss feedback, ArtiBoost enhances training data diversity and model generalization, achieving state-of-the-art performance on HO3D and DexYCB benchmarks even with simple baseline networks.

ABSTRACT

Estimating the articulated 3D hand-object pose from a single RGB image is a highly ambiguous and challenging problem, requiring large-scale datasets that contain diverse hand poses, object types, and camera viewpoints. Most real-world datasets lack these diversities. In contrast, data synthesis can easily ensure those diversities separately. However, constructing both valid and diverse hand-object interactions and efficiently learning from the vast synthetic data is still challenging. To address the above issues, we propose ArtiBoost, a lightweight online data enhancement method. ArtiBoost can cover diverse hand-object poses and camera viewpoints through sampling in a Composited hand-object Configuration and Viewpoint space (CCV-space) and can adaptively enrich the current hard-discernable items by loss-feedback and sample re-weighting. ArtiBoost alternatively performs data exploration and synthesis within a learning pipeline, and those synthetic data are blended into real-world source data for training. We apply ArtiBoost on a simple learning baseline network and witness the performance boost on several hand-object benchmarks. Our models and code are available at https://github.com/lixiny/ArtiBoost.

Motivation & Objective

• To address the challenge of limited diversity in real-world datasets for articulated 3D hand-object pose estimation (HOPE), which suffer from insufficient pose, object, and viewpoint variation.
• To overcome the inefficiency of offline data synthesis by enabling adaptive, online data enrichment that focuses on hard-to-distinguish samples.
• To develop a model-agnostic method that improves HOPE performance through continuous interaction between data exploration and model learning.
• To demonstrate that diverse, physically plausible hand-object interactions synthesized in a structured CCV-space significantly improve pose estimation accuracy.

Proposed method

• The method introduces a three-dimensional discrete space called CCV-space, parameterized by object type, hand pose, and camera viewpoint, to systematically represent hand-object interactions.
• A fitting-based grasp synthesis method enforces contact constraints between MANO hand vertices and object surfaces to generate valid, realistic hand-object configurations in CCV-space.
• ArtiBoost alternates between exploration (sampling triplets from CCV-space) and synthesis (rendering images from sampled views), blending synthetic and real data in training batches.
• Loss feedback from the HOPE model is used to re-weight sampling probabilities, prioritizing hard-to-distinguish configurations in subsequent rounds.
• The framework is model-agnostic and can be integrated with any CNN-based pose estimation architecture, including classification- and regression-based models.
• A lookup table-based exploration strategy is used due to the non-differentiable renderer, enabling efficient and scalable sampling in CCV-space.

Experimental results

Research questions

• RQ1 Can online, loss-guided data exploration in a structured CCV-space improve the generalization of 3D hand-object pose estimation models?
• RQ2 Does synthesizing diverse, physically plausible hand-object interactions in CCV-space lead to better performance than conventional offline synthesis methods?
• RQ3 Can a lightweight, model-agnostic data augmentation framework outperform state-of-the-art methods without architectural modifications?
• RQ4 To what extent does reducing reliance on real-world labeled data improve model performance when combined with online synthetic data enrichment?

Key findings

• The model trained with ArtiBoost's synthetic data in CCV-space outperforms the same model trained with conventional synthetic data from YCBAfford, demonstrating the superiority of CCV-space's diversity and validity.
• The online re-weighting strategy in ArtiBoost leads to faster convergence and higher performance compared to an offline sampling scheme, as shown on the DexYCB benchmark.
• Using only 10% of the original HO3D training data plus 100% of ArtiBoost’s synthetic data achieves better performance than using 100% of the real data alone.
• When ported to a camera-space HOPE framework (Hasson et al.), ArtiBoost significantly improves all metrics, including MPJPE in both camera-space and wrist-relative systems.
• Even with a simple classification-based and regression-based baseline, ArtiBoost enables performance that surpasses previous state-of-the-art results on HO3D (v1-v3) and DexYCB benchmarks.