[Paper Review] Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense

Yixin Chen, Siyuan Huang+4 more|arXiv (Cornell University)|Sep 4, 2019

Human Pose and Action Recognition58 references17 citations

TL;DR

This paper proposes Holistic++ scene understanding, a unified framework for single-view 3D holistic scene parsing and human pose estimation by jointly leveraging human-object interaction (HOI) and physical commonsense reasoning. Using Markov chain Monte Carlo (MCMC) inference on a structured parse graph, the method improves 3D reconstruction accuracy and generalization across datasets by enforcing spatial plausibility and interaction constraints, outperforming baselines on PiGraphs, Watch-n-Patch, and SUN RGB-D in both 2D and 3D metrics.

ABSTRACT

We propose a new 3D holistic++ scene understanding problem, which jointly tackles two tasks from a single-view image: (i) holistic scene parsing and reconstruction---3D estimations of object bounding boxes, camera pose, and room layout, and (ii) 3D human pose estimation. The intuition behind is to leverage the coupled nature of these two tasks to improve the granularity and performance of scene understanding. We propose to exploit two critical and essential connections between these two tasks: (i) human-object interaction (HOI) to model the fine-grained relations between agents and objects in the scene, and (ii) physical commonsense to model the physical plausibility of the reconstructed scene. The optimal configuration of the 3D scene, represented by a parse graph, is inferred using Markov chain Monte Carlo (MCMC), which efficiently traverses through the non-differentiable joint solution space. Experimental results demonstrate that the proposed algorithm significantly improves the performance of the two tasks on three datasets, showing an improved generalization ability.

Motivation & Objective

• To address the challenge of jointly reconstructing 3D scenes and human poses from a single RGB image, leveraging interdependencies between the two tasks.
• To improve 3D scene understanding by incorporating human-object interaction (HOI) as a geometric constraint to refine spatial relations between agents and objects.
• To enhance physical plausibility and stability of 3D reconstructions by modeling physical commonsense, such as support and object permanence.
• To improve generalization across diverse datasets by embedding HOI and physical commonsense as general prior knowledge.
• To develop a non-differentiable, joint inference framework that efficiently explores complex solution spaces using MCMC on a structured parse graph.

Proposed method

• The method represents the 3D scene configuration as a parse graph with a hierarchical parse tree and a Markov random field (MRF) over terminal nodes to model contextual relations.
• Joint inference is performed via maximum a posteriori (MAP) estimation, combining likelihood from 2D detections and projected 3D predictions with priors on HOI and physical commonsense.
• Human-object interaction (HOI) is modeled as a prior term in the energy function, constraining plausible spatial configurations (e.g., sitting, holding) between humans and objects.
• Physical commonsense is enforced via a penalty term that penalizes physically implausible configurations, such as objects floating or penetrating surfaces.
• A Markov chain Monte Carlo (MCMC) sampler traverses the non-differentiable solution space to optimize the parse graph, enabling joint refinement of 3D poses, object boxes, camera pose, and room layout.
• The framework uses a top-down sampling process guided by HOI to improve object detection and pose estimation, especially in complex or occluded scenes.

Experimental results

Research questions

• RQ1 Can joint inference of 3D scene parsing and human pose estimation improve performance beyond isolated task optimization?
• RQ2 How effective is human-object interaction (HOI) reasoning in reducing geometric ambiguity and improving spatial accuracy in single-view 3D reconstruction?
• RQ3 To what extent does incorporating physical commonsense enhance the plausibility and stability of 3D scene reconstructions?
• RQ4 Can HOI and physical commonsense serve as general priors to improve model generalization across diverse datasets with varying real-world complexity?
• RQ5 How does the MCMC-based inference framework handle the non-differentiable, high-dimensional solution space of holistic 3D scene understanding?

Key findings

• The proposed method achieves state-of-the-art performance on 3D human pose estimation, reducing 3D error from 0.732m (VNect) to 0.472m on PiGraphs and from 0.646m (VNect) to 0.330m on Watch-n-Patch.
• On the synthetic SUN RGB-D dataset, the method reduces 3D pose error from 0.435m (baseline) to 0.517m, demonstrating robustness despite using ground-truth 2D poses.
• Ablative analysis shows that removing HOI reduces 3D object IoU by 0.9% (to 27.8%) and increases 3D pose error by 0.069m (to 0.521m), indicating HOI's critical role in spatial reasoning.
• The full model reduces physical violations (distance between object and support surface) from 0.223m (w/o phy.) to 0.150m, confirming improved physical plausibility.
• The model achieves a 13.1% miss-detection rate for HOI interactions (e.g., sitting), significantly lower than the 15.2% rate when HOI is excluded, showing improved detection via top-down reasoning.
• Qualitative results demonstrate consistent generalization across PiGraphs, Watch-n-Patch, and SUN RGB-D, with improved initialization and physically plausible 3D layouts.