[Paper Review] Generating 3D People in Scenes without People

Yan Zhang, Mohamed S. Hassan+3 more|arXiv (Cornell University)|Dec 5, 2019

Human Pose and Action Recognition56 references4 citations

TL;DR

This paper presents a two-stage generative framework that automatically synthesizes semantically and physically plausible 3D human bodies in people-free 3D scenes. It uses a conditional variational autoencoder (CVAE) conditioned on scene depth and semantics to generate diverse, shape-aware human meshes, followed by geometry-aware fitting to enforce physical plausibility, significantly outperforming prior methods in realism, diversity, and interaction fidelity.

ABSTRACT

We present a fully automatic system that takes a 3D scene and generates plausible 3D human bodies that are posed naturally in that 3D scene. Given a 3D scene without people, humans can easily imagine how people could interact with the scene and the objects in it. However, this is a challenging task for a computer as solving it requires that (1) the generated human bodies to be semantically plausible within the 3D environment (e.g. people sitting on the sofa or cooking near the stove), and (2) the generated human-scene interaction to be physically feasible such that the human body and scene do not interpenetrate while, at the same time, body-scene contact supports physical interactions. To that end, we make use of the surface-based 3D human model SMPL-X. We first train a conditional variational autoencoder to predict semantically plausible 3D human poses conditioned on latent scene representations, then we further refine the generated 3D bodies using scene constraints to enforce feasible physical interaction. We show that our approach is able to synthesize realistic and expressive 3D human bodies that naturally interact with 3D environment. We perform extensive experiments demonstrating that our generative framework compares favorably with existing methods, both qualitatively and quantitatively. We believe that our scene-conditioned 3D human generation pipeline will be useful for numerous applications; e.g. to generate training data for human pose estimation, in video games and in VR/AR. Our project page for data and code can be seen at: \url{https://vlg.inf.ethz.ch/projects/PSI/}.

Motivation & Objective

• To address the lack of realistic, automatically generated 3D human bodies in 3D scenes, which limits applications in VR/AR, robotics, and synthetic data generation.
• To model semantically plausible human-scene interactions where humans are positioned meaningfully relative to scene objects (e.g., sitting on sofas, standing near stoves).
• To ensure physically feasible interactions by preventing body interpenetration and enforcing surface contact, while preserving natural poses.
• To develop a scene-conditioned 3D human generation pipeline that can be used as a prior for downstream tasks like monocular 3D human pose estimation.
• To establish a benchmark with new datasets and evaluation metrics for scene-aware 3D human mesh generation.

Proposed method

• A conditional variational autoencoder (CVAE) is trained to generate 3D human body meshes conditioned on latent representations of scene depth and semantics.
• The SMPL-X model is used as the underlying body representation, enabling detailed, differentiable 3D mesh output with shape and pose parameters.
• Scene geometry-aware fitting is applied post-generation to refine poses, minimizing interpenetration and ensuring contact with scene surfaces.
• The method incorporates a loss term $\mathcal{L}_f$ to enforce physical plausibility by penalizing collisions and floating.
• The framework is trained on an extended version of the PROX-Qualitative dataset, with synthetic data generated via rendering from multiple virtual viewpoints.
• The approach is evaluated using three metrics: semantic plausibility, physical plausibility, and diversity, with user studies and quantitative benchmarks.

Experimental results

Research questions

• RQ1 Can a learned generative model produce diverse, semantically plausible 3D human poses in 3D scenes without human supervision?
• RQ2 How effectively can scene geometry-aware fitting correct physically implausible configurations such as interpenetration or floating?
• RQ3 To what extent does the scene-conditioned prior improve downstream 3D human pose estimation from monocular RGB images?
• RQ4 How does the proposed method compare quantitatively and qualitatively to state-of-the-art baselines in 3D human mesh generation?
• RQ5 Can the method generalize across diverse indoor scenes and produce realistic human-body interactions?

Key findings

• The proposed method generates 3D human bodies with significantly higher diversity than the baseline, as shown by a 25% increase in diversity score (Tab. 2), while maintaining comparable semantic plausibility.
• User studies show that the method's results are rated 3.49/5.0 on average for PROX-E and 3.30/5.0 for MP3D-R, outperforming the baseline and improving upon the baseline with added physical loss.
• The method improves 3D human pose estimation on the PROX-Quantitative dataset, reducing mean per-joint error (PJE) to 174.10 mm and vertex-to-vertex error (V2V) to 171.75 mm, outperforming Simplify-X and matching PROX.
• The geometry-aware fitting step reduces interpenetration and improves physical plausibility, as evidenced by lower collision penalties and improved contact consistency in qualitative results.
• The scene-conditioned prior derived from the model improves 3D pose estimation by providing a scene-aware initialization, reducing error by up to 1.5% in V2V error compared to standard baselines.
• The method achieves state-of-the-art performance in generating realistic, diverse, and physically plausible human bodies in 3D scenes, validated through comprehensive quantitative and qualitative evaluation.