[Paper Review] SeqHAND:RGB-Sequence-Based 3D Hand Pose and Shape Estimation

John Yang, Hyung Jin Chang+2 more|arXiv (Cornell University)|Jul 10, 2020

Human Pose and Action Recognition50 references4 citations

TL;DR

This paper proposes SeqHAND, a novel RGB-sequence-based 3D hand pose and shape estimation framework that leverages temporal motion dynamics through a synthetic sequential dataset and a recurrent ConvLSTM-based network. By generating pose-flows from static annotations and preserving visuo-temporal features during synthetic-to-real domain adaptation, the method achieves state-of-the-art performance with smoother, more natural hand motion estimations compared to frame-by-frame baselines.

ABSTRACT

3D hand pose estimation based on RGB images has been studied for a long time. Most of the studies, however, have performed frame-by-frame estimation based on independent static images. In this paper, we attempt to not only consider the appearance of a hand but incorporate the temporal movement information of a hand in motion into the learning framework for better 3D hand pose estimation performance, which leads to the necessity of a large scale dataset with sequential RGB hand images. We propose a novel method that generates a synthetic dataset that mimics natural human hand movements by re-engineering annotations of an extant static hand pose dataset into pose-flows. With the generated dataset, we train a newly proposed recurrent framework, exploiting visuo-temporal features from sequential images of synthetic hands in motion and emphasizing temporal smoothness of estimations with a temporal consistency constraint. Our novel training strategy of detaching the recurrent layer of the framework during domain finetuning from synthetic to real allows preservation of the visuo-temporal features learned from sequential synthetic hand images. Hand poses that are sequentially estimated consequently produce natural and smooth hand movements which lead to more robust estimations. We show that utilizing temporal information for 3D hand pose estimation significantly enhances general pose estimations by outperforming state-of-the-art methods in experiments on hand pose estimation benchmarks.

Motivation & Objective

• To address the scarcity of large-scale, realistic RGB sequential hand image datasets for 3D hand pose estimation.
• To improve 3D hand pose estimation by incorporating temporal motion dynamics beyond static image appearance.
• To reduce the synthetic-to-real domain gap in hand pose estimation by preserving visuo-temporal features during domain adaptation.
• To develop a recurrent deep learning framework that directly exploits sequential RGB inputs for joint 3D pose and shape estimation.
• To enhance robustness to occlusions and improve temporal consistency in hand motion estimation.

Proposed method

• Proposes a pose-flow generation method that re-engineers static hand pose annotations from the BigHand2.2M dataset into temporally coherent sequences of synthetic RGB hand images.
• Introduces a recurrent encoder-decoder framework with ConvLSTM layers to model visuo-temporal dependencies in image sequences for 3D hand pose and shape prediction.
• Employs a temporal consistency loss that penalizes large jumps between consecutive pose estimations, enforcing smooth motion trajectories.
• Designs a novel domain adaptation strategy that detaches the recurrent layer during finetuning from synthetic to real data, preserving learned temporal features.
• Utilizes a multi-stage training pipeline: pre-train on synthetic SeqHAND dataset, then finetune on real data while keeping the ConvLSTM layer frozen to maintain temporal representation.
• Leverages a 3D hand model (MANO) to regress 3D joint locations and shape parameters from image features.

Experimental results

Research questions

• RQ1 Can temporal motion information from sequential RGB images significantly improve 3D hand pose estimation accuracy compared to frame-by-frame methods?
• RQ2 Can a synthetic dataset with realistic pose-flows effectively train a deep learning model for generalization to real-world RGB sequences?
• RQ3 Does preserving visuo-temporal features during synthetic-to-real domain adaptation lead to better performance than end-to-end finetuning?
• RQ4 Can temporal modeling improve robustness to occlusions and self-occlusions in 3D hand pose estimation?
• RQ5 Can a recurrent network trained on synthetic sequences produce more natural and smooth hand motion trajectories than appearance-only baselines?

Key findings

• The proposed method achieves state-of-the-art performance on the STB, ED, and DO benchmarks, with a 3D-PCK@10 threshold of 89.7% on the ED dataset and 85.3% on the DO dataset.
• By detaching the ConvLSTM layer during real-domain finetuning, the model preserves visuo-temporal features, leading to a 5.2% absolute improvement in 3D-PCK over baseline finetuning.
• The average temporal difference in shape parameters (β) is 4.16×10⁻¹¹ for SeqHAND-Net versus 2.38×10⁻⁵ for frame-by-frame estimation, indicating significantly smoother shape evolution.
• The average pose parameter difference (θ) between frames is 1.88×10⁻⁶ for SeqHAND-Net versus 6.90×10⁻⁶ for the frame-by-frame method, confirming enhanced temporal smoothness.
• The model demonstrates superior robustness to occlusions, particularly on the DO dataset with heavy occlusions, outperforming methods relying on external 2D pose estimators.
• Qualitative results show that SeqHAND-Net produces natural, flowing hand motion trajectories, while frame-by-frame methods exhibit unnatural jumps and inconsistent postures across frames.