[Paper Review] GRAINS: Generative Recursive Autoencoders for INdoor Scenes

Manyi Li, Akshay Gadi Patil+8 more|arXiv (Cornell University)|Jul 24, 2018

3D Shape Modeling and Analysis36 references50 citations

TL;DR

GRAINS is a generative recursive autoencoder (RvNN-VAE) that learns hierarchical indoor scene structures and rapidly generates plausible 3D indoor scenes by decoding random Gaussian codes into scene hierarchies of labeled OBBs, then retrieving 3D objects.

ABSTRACT

We present a generative neural network which enables us to generate plausible 3D indoor scenes in large quantities and varieties, easily and highly efficiently. Our key observation is that indoor scene structures are inherently hierarchical. Hence, our network is not convolutional; it is a recursive neural network or RvNN. Using a dataset of annotated scene hierarchies, we train a variational recursive autoencoder, or RvNN-VAE, which performs scene object grouping during its encoding phase and scene generation during decoding. Specifically, a set of encoders are recursively applied to group 3D objects based on support, surround, and co-occurrence relations in a scene, encoding information about object spatial properties, semantics, and their relative positioning with respect to other objects in the hierarchy. By training a variational autoencoder (VAE), the resulting fixed-length codes roughly follow a Gaussian distribution. A novel 3D scene can be generated hierarchically by the decoder from a randomly sampled code from the learned distribution. We coin our method GRAINS, for Generative Recursive Autoencoders for INdoor Scenes. We demonstrate the capability of GRAINS to generate plausible and diverse 3D indoor scenes and compare with existing methods for 3D scene synthesis. We show applications of GRAINS including 3D scene modeling from 2D layouts, scene editing, and semantic scene segmentation via PointNet whose performance is boosted by the large quantity and variety of 3D scenes generated by our method.

Motivation & Objective

• Motivate the need for scalable, realistic 3D indoor scene generation for VR/AR, robotics, and scene understanding.
• Propose a hierarchical, structure-aware generative model (RvNN-VAE) tailored to indoor scenes.
• Encode object semantics and spatial relations via hierarchical OBB representations to enable diverse scene synthesis.
• Enable downstream tasks such as 3D scene modeling from layouts, editing, and semantic segmentation enhancements.

Proposed method

• Represent indoor scenes as labeled oriented bounding boxes (OBBs) organized into a hierarchy.
• Train a variational recursive autoencoder (RvNN-VAE) with multiple encoder/decoder pairs for different relations (BoxEnc, SuppEnc, Co-ocEnc, SurrEnc, WallEnc, RootEnc).
• Encode relative 2D positioning and object semantics in a 28-D vector per internal node, including rotation, offsets, attachments, and alignments.
• Decode from a root code sampled from a Gaussian distribution to a scene hierarchy, then replace OBBs with 3D objects from a database.
• Use a Node Classifier to select the appropriate decoder at each recursion during generation.
• Ground learning with perceptual studies and comparisons to existing 3D scene synthesis methods.

Experimental results

Research questions

• RQ1 Can a hierarchical, structure-aware generative model produce realistic and diverse 3D indoor scenes at scale?
• RQ2 How do relative positioning and semantic co-occurrence encoding impact scene plausibility and diversity?
• RQ3 How does GRAINS compare with existing 3D scene synthesis methods in terms of quality and variety?
• RQ4 What practical applications (layout-from-2D, editing, segmentation) benefit from large, varied synthetic scene data?

Key findings

• GRAINS can generate a large number of plausible and diverse 3D indoor scenes in a fraction of a second per scene.
• Plausibility validated by perceptual studies and comparative evaluations against existing baseline methods.
• Experiments validate the effectiveness of semantic labels and relative positioning in improving generation quality.
• Generated scenes boost performance of downstream tasks such as semantic segmentation via PointNet and enable 3D scene modeling from 2D layouts.
• Hierarchical encoding and the three relation types (support, surround, co-occurrence) are key design choices that improve learnability and realism.