[Paper Review] DECO: Dense Estimation of 3D Human-Scene Contact In The Wild

Shashank Tripathi, Agniv Chatterjee+4 more|UvA-DARE (University of Amsterdam)|Sep 26, 2023

Human Pose and Action RecognitionComputer Science3 citations

TL;DR

DECO proposes a novel 3D contact estimation method for dense, vertex-level human-body contact with objects and scenes in unconstrained, in-the-wild images. It leverages a new dataset, DAMON, with human-annotated 3D contact labels and a dual-attention network combining body-part-driven and scene-context-driven reasoning to achieve state-of-the-art performance on RICH, BEHAVE, and DAMON benchmarks, outperforming prior methods significantly.

ABSTRACT

Understanding how humans use physical contact to interact with the world is key to enabling human-centric artificial intelligence. While inferring 3D contact is crucial for modeling realistic and physically-plausible human-object interactions, existing methods either focus on 2D, consider body joints rather than the surface, use coarse 3D body regions, or do not generalize to in-the-wild images. In contrast, we focus on inferring dense, 3D contact between the full body surface and objects in arbitrary images. To achieve this, we first collect DAMON, a new dataset containing dense vertex-level contact annotations paired with RGB images containing complex human-object and human-scene contact. Second, we train DECO, a novel 3D contact detector that uses both body-part-driven and scene-context-driven attention to estimate vertex-level contact on the SMPL body. DECO builds on the insight that human observers recognize contact by reasoning about the contacting body parts, their proximity to scene objects, and the surrounding scene context. We perform extensive evaluations of our detector on DAMON as well as on the RICH and BEHAVE datasets. We significantly outperform existing SOTA methods across all benchmarks. We also show qualitatively that DECO generalizes well to diverse and challenging real-world human interactions in natural images. The code, data, and models are available at https://deco.is.tue.mpg.de.

Motivation & Objective

• To enable accurate, dense, 3D contact estimation between full human body surfaces and scene objects in unconstrained, in-the-wild images.
• To address the lack of high-quality, dense 3D contact annotations in real-world images, which limits training of robust 3D contact detectors.
• To develop a method that reasons about both body-part identity and surrounding scene context to infer contact, improving generalization beyond 2D or coarse 3D representations.
• To create a new benchmark dataset, DAMON, with human-annotated, vertex-level 3D contact labels on SMPL body meshes for diverse human-object and human-scene interactions.
• To outperform existing state-of-the-art methods in 3D contact estimation across multiple benchmarks, including in-the-wild and scanned datasets.

Proposed method

• Proposes DECO, a 3D contact detector that takes a single RGB image as input and outputs dense contact labels on a SMPL body mesh at the vertex level.
• Employs a dual-attention mechanism: body-part-driven attention to focus on relevant body regions and scene-context-driven attention to reason about proximity and spatial relationships with objects.
• Trains the model end-to-end using a multi-task loss combining contact classification, part-aware loss, and scene-aware supervision, with empirically tuned loss weights.
• Uses a 2D feature extractor (ResNet-based) to process input images, scene segmentation masks, and part segmentation masks, followed by feature fusion and 3D regression.
• Introduces context branches (L²_s and L²_p) that refine predictions using 2D scene and part context features, improving geodesic error by ~24%.
• Leverages a novel interactive 3D labeling tool to collect the DAMON dataset, enabling precise, human-annotated 3D contact labels on full-body meshes.

Experimental results

Research questions

• RQ1 Can a deep learning model accurately predict dense, 3D contact between a full human body and scene objects in unconstrained, real-world images?
• RQ2 How does integrating both body-part identity and scene context improve contact estimation compared to methods relying on only one modality?
• RQ3 To what extent does a human-annotated 3D contact dataset (DAMON) improve generalization and accuracy compared to synthetic or SDF-based pseudo-ground-truths?
• RQ4 Does the proposed dual-attention mechanism outperform existing architectures in capturing complex, full-body contact patterns in diverse interactions?
• RQ5 How does DECO generalize to out-of-distribution, internet-sourced images not seen during training?

Key findings

• DECO achieves a mean IoU of 0.510 on the RICH test set and 0.450 mIoU on the DAMON dataset, significantly outperforming prior SOTA methods.
• On the RICH dataset, DECO achieves a 24% improvement in geodesic error over the base model by incorporating context branches.
• The DAMON dataset achieves a Fleiss’ Kappa of 0.656, indicating substantial agreement among annotators, and a label accuracy mIoU of 0.450, with human-annotated labels showing superior fidelity to real contact compared to SDF-based pseudo-gt.
• Qualitative results on internet images show DECO generalizes well to diverse, challenging real-world interactions beyond the training distribution.
• The model with context branches adds only 1% complexity but improves performance significantly, demonstrating the value of scene and part context in contact reasoning.
• DECO outperforms baseline methods like BSTRO and POSA^PIXIE across all benchmarks, including on the RICH and BEHAVE datasets.