[Paper Review] ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstruction

Zhengdi Yu, Shaoli Huang+3 more|arXiv (Cornell University)|Mar 10, 2023

Artificial Intelligence in Healthcare and EducationMedicine3 citations

TL;DR

ACR proposes an attention collaboration-based regressor for arbitrary two-hand 3D reconstruction from monocular RGB images, using center and part-based attention to disentangle hand representations and reduce sensitivity to occlusion and truncation. It outperforms state-of-the-art interacting-hand methods on InterHand2.6M (8.49 MPJPE) and achieves comparable performance to single-hand SOTA on FreiHand, demonstrating robustness in real-world and challenging scenarios.

ABSTRACT

Reconstructing two hands from monocular RGB images is challenging due to frequent occlusion and mutual confusion. Existing methods mainly learn an entangled representation to encode two interacting hands, which are incredibly fragile to impaired interaction, such as truncated hands, separate hands, or external occlusion. This paper presents ACR (Attention Collaboration-based Regressor), which makes the first attempt to reconstruct hands in arbitrary scenarios. To achieve this, ACR explicitly mitigates interdependencies between hands and between parts by leveraging center and part-based attention for feature extraction. However, reducing interdependence helps release the input constraint while weakening the mutual reasoning about reconstructing the interacting hands. Thus, based on center attention, ACR also learns cross-hand prior that handle the interacting hands better. We evaluate our method on various types of hand reconstruction datasets. Our method significantly outperforms the best interacting-hand approaches on the InterHand2.6M dataset while yielding comparable performance with the state-of-the-art single-hand methods on the FreiHand dataset. More qualitative results on in-the-wild and hand-object interaction datasets and web images/videos further demonstrate the effectiveness of our approach for arbitrary hand reconstruction. Our code is available at https://github.com/ZhengdiYu/Arbitrary-Hands-3D-Reconstruction.

Motivation & Objective

• To address the challenge of 3D hand reconstruction in arbitrary scenarios, including occlusion, truncation, and mutual confusion.
• To reduce dependency between hands and body parts by disentangling representations, thereby improving robustness to incomplete or impaired inputs.
• To maintain strong interaction modeling between hands despite reduced interdependence, ensuring accurate reconstruction of interacting hands.
• To develop a method that generalizes beyond constrained two-hand inputs, enabling practical deployment in real-world settings like in-the-wild images and videos.

Proposed method

• ACR employs an Attention Encoder (AE) that generates hand-center and per-part attention maps using a cross-hand prior to estimate visibility and guide feature learning.
• It uses a novel Attention Collaboration-based Feature Aggregator (ACFA) that fuses global (G), part-based (P), and cross-hand-attention prior (C) representations for collaborative hand regression.
• The cross-hand prior reasoning module with an interaction field dynamically adjusts dependency strength between hands, enhancing modeling of mutual interactions.
• The method is end-to-end trainable and does not require hand bounding boxes or ground-truth scale, enabling deployment in real-world scenarios.
• It leverages a parametric hand model (e.g., MANO) for 3D mesh regression, with supervision from 3D keypoint and mesh annotations.
• The network is trained using standard 3D hand reconstruction loss (e.g., MPJPE, MPVPE) on multi-scenario datasets.

Experimental results

Research questions

• RQ1 Can a 3D hand reconstruction method generalize to arbitrary hand configurations, including truncated, separate, or occluded hands, without requiring two-hand interaction as input?
• RQ2 How can interdependencies between hands and between hand parts be explicitly mitigated to improve robustness to partial occlusion and truncation?
• RQ3 To what extent can cross-hand prior reasoning compensate for reduced interdependence and maintain accurate interaction modeling in complex scenarios?
• RQ4 How does the proposed attention collaboration mechanism compare to unified feature learning in terms of performance and generalization on diverse datasets?

Key findings

• ACR achieves the lowest MPJPE of 8.49 on the InterHand2.6M interacting-hand (IH) subset, outperforming the previous SOTA method IntagHand.
• On the FreiHand single-hand benchmark, ACR achieves 6.91 MPJPE, comparable to state-of-the-art single-hand methods, demonstrating strong generalization.
• Ablation studies confirm that both part-based and cross-hand prior representations significantly improve performance, with the cross-hand prior showing greater gains on interacting-hand scenarios.
• Qualitative results on in-the-wild images, videos, and hand-object interaction datasets show accurate reconstruction even under severe occlusion and non-standard poses.
• The method is robust to input variations such as single hands, ego-views, and truncated hands, proving its practicality for real-world applications.
• The model achieves a 5.21 PAMPJPE on the full InterHand2.6M dataset when combining global, part, and cross-hand prior representations, indicating strong overall performance.