[Paper Review] Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments

Xueting Li, Sifei Liu+4 more|arXiv (Cornell University)|Mar 13, 2019

Human Pose and Action Recognition31 references4 citations

TL;DR

This paper proposes a two-stage framework for 3D human affordance prediction in indoor scenes, combining a fully automatic 3D pose synthesizer that fuses 2D pose distributions from TV shows with 3D geometric constraints from voxelized scenes, and a generative model that predicts semantically plausible and physically feasible 3D human poses from a single RGB/RGB-D image. The method achieves state-of-the-art performance, with a 27.6% improvement in geometry score over prior work.

ABSTRACT

Affordance modeling plays an important role in visual understanding. In this paper, we aim to predict affordances of 3D indoor scenes, specifically what human poses are afforded by a given indoor environment, such as sitting on a chair or standing on the floor. In order to predict valid affordances and learn possible 3D human poses in indoor scenes, we need to understand the semantic and geometric structure of a scene as well as its potential interactions with a human. To learn such a model, a large-scale dataset of 3D indoor affordances is required. In this work, we build a fully automatic 3D pose synthesizer that fuses semantic knowledge from a large number of 2D poses extracted from TV shows as well as 3D geometric knowledge from voxel representations of indoor scenes. With the data created by the synthesizer, we introduce a 3D pose generative model to predict semantically plausible and physically feasible human poses within a given scene (provided as a single RGB, RGB-D, or depth image). We demonstrate that our human affordance prediction method consistently outperforms existing state-of-the-art methods.

Motivation & Objective

• To address the lack of large-scale, physically feasible 3D human pose annotations in indoor scenes for affordance learning.
• To overcome the limitations of 2D-based methods that fail to enforce 3D geometric plausibility in human-scene interactions.
• To develop a fully automatic 3D pose synthesizer that leverages semantic knowledge from 2D poses and geometric constraints from 3D voxelized scenes.
• To train an end-to-end generative model that jointly predicts plausible 3D pose locations and appearances conditioned on a single scene image.
• To establish a new benchmark for 3D affordance prediction on the SUNCG dataset using synthesized, physically valid human poses.

Proposed method

• A fully automatic 3D pose synthesizer is developed by transferring 2D human poses from the Sitcom dataset to 3D voxelized scenes in SUNCG using camera parameters and 3D geometry.
• The synthesized 3D poses are refined via collision detection and support checking to ensure physical feasibility, eliminating intersections and ensuring proper body support.
• A two-stream generative model is proposed, with a 'where' module predicting pelvis joint locations and a 'what' module generating full 3D pose appearances conditioned on location.
• The model uses a differentiable geometry-aware discriminator to enforce 3D physical consistency, even when input is a single RGB image.
• The pose distribution is factorized into location and appearance components, with the pelvis location serving as a differentiable bridge between modules.
• The framework is trained end-to-end using a combination of perceptual, geometry, and adversarial losses to improve realism and physical plausibility.

Experimental results

Research questions

• RQ1 Can a fully automatic 3D pose synthesizer generate large-scale, physically feasible human poses by fusing 2D pose distributions with 3D scene geometry?
• RQ2 How can a generative model predict semantically plausible and geometrically valid 3D human poses from a single RGB or RGB-D image?
• RQ3 To what extent does incorporating 3D geometric reasoning improve the physical feasibility of predicted human poses compared to 2D-only baselines?
• RQ4 Can the model hallucinate depth and 3D structure from a single 2D image to generate contextually appropriate poses at varying depths?
• RQ5 What is the impact of geometry-aware supervision on the realism and correctness of generated 3D human poses?

Key findings

• The proposed method achieves a 78.43% geometry score on the what module using RGB-D input, a 27.6% improvement over the baseline method’s 56.29% score.
• With only RGB input, the model still achieves a 6.66% higher geometry score than the baseline, demonstrating strong 3D geometry hallucination from 2D images.
• The model successfully generates poses that vary in scale and action (e.g., standing to sitting) when depth is interpolated from d-0.5 to d+0.5, showing depth-aware pose generation.
• User studies and a trained authenticity classifier confirm that generated poses are more realistic and physically plausible than those from prior methods.
• Failure cases reveal that errors primarily stem from semantic misalignment (e.g., sitting on cabinets) or geometric violations (e.g., collisions), indicating challenges in 3D reasoning from 2D inputs.
• The geometry-aware discriminator significantly improves physical plausibility, reducing geometry score drop by 4.59% when RGB input is used without it.