[Paper Review] GRASS: Generative Recursive Autoencoders for Shape Structures

Jun Li, Kai Xu+4 more|arXiv (Cornell University)|May 5, 2017

3D Shape Modeling and Analysis63 references122 citations

TL;DR

GRASS introduces a Generative Recursive Autoencoder for shape structures that encodes hierarchies of shape parts and synthesizes new structures via a GAN, followed by geometry synthesis for parts.

ABSTRACT

We introduce a novel neural network architecture for encoding and synthesis of 3D shapes, particularly their structures. Our key insight is that 3D shapes are effectively characterized by their hierarchical organization of parts, which reflects fundamental intra-shape relationships such as adjacency and symmetry. We develop a recursive neural net (RvNN) based autoencoder to map a flat, unlabeled, arbitrary part layout to a compact code. The code effectively captures hierarchical structures of man-made 3D objects of varying structural complexities despite being fixed-dimensional: an associated decoder maps a code back to a full hierarchy. The learned bidirectional mapping is further tuned using an adversarial setup to yield a generative model of plausible structures, from which novel structures can be sampled. Finally, our structure synthesis framework is augmented by a second trained module that produces fine-grained part geometry, conditioned on global and local structural context, leading to a full generative pipeline for 3D shapes. We demonstrate that without supervision, our network learns meaningful structural hierarchies adhering to perceptual grouping principles, produces compact codes which enable applications such as shape classification and partial matching, and supports shape synthesis and interpolation with significant variations in topology and geometry.

Motivation & Objective

• Motivate learning a generative model for structured 3D shapes that capture part hierarchy and symmetry.
• Develop a recursive autoencoder that encodes and decodes shape part layouts into fixed-length root codes.
• Extend the autoencoder with a GAN to learn a plausible manifold of shape structures.
• Add a second network to convert synthesized structure boxes into detailed part geometries.
• Demonstrate unsupervised discovery of meaningful hierarchical structures and enable synthesis and interpolation.

Proposed method

• Represent a 3D shape as a symmetry hierarchy of oriented bounding boxes (OBBs).
• Use a recursive autoencoder with two node types (adjacency and symmetry) to encode/decode hierarchies into fixed-length codes.
• Train a NodeClsfr to select the appropriate decoder (AdjDec, SymDec, BoxDec) at each step.
• Train a GAN on root codes to learn a low-dimensional manifold of plausible structures and sample from it to synthesize new layouts.
• Use a second network to map synthesized boxes to voxel-based part geometries conditioned on context.
• Pretrain the autoencoder unsupervised on randomly generated hierarchies, then adversarially tune the decoder for generative sampling.

Experimental results

Research questions

• RQ1 Can a recursive autoencoder learn meaningful hierarchical structure and symmetry in unlabeled 3D shape parts?
• RQ2 Does learning a manifold over root codes via GAN enable plausible generation and interpolation of shape structures?
• RQ3 Can the synthesized structure boxes be effectively converted into detailed part geometries to form complete 3D shapes?
• RQ4 Is the learned structure robust to variations in topology and geometry across object categories (e.g., chairs)?

Key findings

• The model learns meaningful hierarchical structures that align with perceptual grouping principles.
• Root codes provide compact representations enabling shape classification and partial matching.
• A GAN over root codes yields a generative model that samples plausible structures and supports interpolation.
• A separate module effectively maps synthesized structures to detailed part geometries for full shape synthesis.
• The approach demonstrates the ability to synthesize shapes with significant topology and geometry variation in an unsupervised setting.