[Paper Review] Detailed 2D-3D Joint Representation for Human-Object Interaction

Yong–Lu Li, Xinpeng Liu+5 more|arXiv (Cornell University)|Apr 17, 2020

Human Pose and Action RecognitionComputer Science61 references17 citations

TL;DR

This paper proposes DJ-RN, a novel 2D-3D joint representation learning framework for Human-Object Interaction (HOI) detection that leverages detailed 3D body shapes (face and hands) and estimated 3D object locations via single-view reconstruction and perspective projection. By fusing 2D and 3D features through cross-modal consistency tasks, it achieves state-of-the-art performance on HICO-DET and a new ambiguous benchmark, Ambiguous-HOI, with 21.34 mAP and 10.37 mAP respectively.

ABSTRACT

Human-Object Interaction (HOI) detection lies at the core of action understanding. Besides 2D information such as human/object appearance and locations, 3D pose is also usually utilized in HOI learning since its view-independence. However, rough 3D body joints just carry sparse body information and are not sufficient to understand complex interactions. Thus, we need detailed 3D body shape to go further. Meanwhile, the interacted object in 3D is also not fully studied in HOI learning. In light of these, we propose a detailed 2D-3D joint representation learning method. First, we utilize the single-view human body capture method to obtain detailed 3D body, face and hand shapes. Next, we estimate the 3D object location and size with reference to the 2D human-object spatial configuration and object category priors. Finally, a joint learning framework and cross-modal consistency tasks are proposed to learn the joint HOI representation. To better evaluate the 2D ambiguity processing capacity of models, we propose a new benchmark named Ambiguous-HOI consisting of hard ambiguous images. Extensive experiments in large-scale HOI benchmark and Ambiguous-HOI show impressive effectiveness of our method. Code and data are available at https://github.com/DirtyHarryLYL/DJ-RN.

Motivation & Objective

• Address the limitations of 2D-based HOI detection, which suffers from viewpoint ambiguity due to sparse 2D pose and appearance cues.
• Overcome the insufficiency of rough 3D joint representations by incorporating detailed 3D human body shapes (face and hands) for richer geometric and semantic understanding.
• Introduce a 3D representation of interacted objects by estimating their location and size using 2D spatial configuration and category priors.
• Develop a joint 2D-3D learning framework that enforces cross-modal consistency to improve disambiguation and robustness.
• Propose a new benchmark, Ambiguous-HOI, to rigorously evaluate models' ability to resolve 2D viewpoint ambiguities in HOI detection.

Proposed method

• Utilize single-view human body capture (e.g., SMPLify-X) to reconstruct detailed 3D human shapes, including face and hand geometry.
• Estimate 3D object location and size using 2D human-object spatial relationships and object category priors, represented as hollow spheres in a normalized 3D volume.
• Construct a 3D HOI spatial configuration volume by placing the 3D human and object representations in a shared 3D space.
• Design a Dual-Stream Network (DJ-RN) with a 2D-RN and a 3D-RN to extract modality-specific features from RGB images and the 3D volume.
• Enforce cross-modal consistency via three tasks: 2D feature alignment guided by 3D spatial features, semantic alignment between 2D and 3D features, and joint attention learning for body parts.
• Apply multi-task learning with loss functions: $γ_{att}$ for attention consistency, $γ_{tri}$ for triplet contrastive learning, and $γ_{sem}$ for semantic consistency.

Experimental results

Research questions

• RQ1 Can detailed 3D human body representations (face and hands) significantly improve HOI detection performance compared to coarse 3D joints?
• RQ2 How effective is 3D object estimation from single-view 2D cues and category priors in enhancing HOI understanding?
• RQ3 To what extent does joint 2D-3D representation learning with cross-modal consistency improve disambiguation in viewpoint-ambiguous images?
• RQ4 Can a new benchmark like Ambiguous-HOI effectively evaluate and differentiate models' robustness to 2D ambiguities?
• RQ5 Does the proposed joint learning framework generalize better to rare and long-tail HOI categories than standalone 2D or 3D models?

Key findings

• DJ-RN achieves 21.34 mAP on the HICO-DET Default Full set, outperforming the previous SoTA method by 1.94 mAP.
• On the proposed Ambiguous-HOI benchmark, DJ-RN achieves 10.37 mAP, surpassing the next best method (Julia et al.) by 0.65 mAP.
• The 3D-RN component alone achieves 12.41 mAP on HICO-DET, showing stronger generalization on rare categories (18.53 mAP) than non-rare ones (22.18 mAP), indicating lower data dependency.
• Ablation studies show that removing hand and face details degrades performance by 0.51 and 0.32 mAP respectively, confirming their importance for hand-related actions.
• The joint learning framework with all consistency losses ($\mathcal{L}_{att}$, $\mathcal{L}_{tri}$, $\mathcal{L}_{sem}$) achieves 21.34 mAP, while removing any one reduces performance by 0.5–0.6 mAP.
• Visualizations confirm that 2D and 3D attention maps are well-aligned and focus on semantically relevant body parts (e.g., hands for 'cut', face for 'talk'), validating the consistency learning.