[Paper Review] StructureNet: Hierarchical Graph Networks for 3D Shape Generation

Kaichun Mo, Paul Guerrero+5 more|arXiv (Cornell University)|Aug 1, 2019

3D Shape Modeling and AnalysisEngineering81 references60 citations

TL;DR

StructureNet introduces a hierarchical graph network to encode n-ary shape part hierarchies with horizontal relations, enabling generation, interpolation, and structure-aware processing of 3D shapes.

ABSTRACT

The ability to generate novel, diverse, and realistic 3D shapes along with associated part semantics and structure is central to many applications requiring high-quality 3D assets or large volumes of realistic training data. A key challenge towards this goal is how to accommodate diverse shape variations, including both continuous deformations of parts as well as structural or discrete alterations which add to, remove from, or modify the shape constituents and compositional structure. Such object structure can typically be organized into a hierarchy of constituent object parts and relationships, represented as a hierarchy of n-ary graphs. We introduce StructureNet, a hierarchical graph network which (i) can directly encode shapes represented as such n-ary graphs; (ii) can be robustly trained on large and complex shape families; and (iii) can be used to generate a great diversity of realistic structured shape geometries. Technically, we accomplish this by drawing inspiration from recent advances in graph neural networks to propose an order-invariant encoding of n-ary graphs, considering jointly both part geometry and inter-part relations during network training. We extensively evaluate the quality of the learned latent spaces for various shape families and show significant advantages over baseline and competing methods. The learned latent spaces enable several structure-aware geometry processing applications, including shape generation and interpolation, shape editing, or shape structure discovery directly from un-annotated images, point clouds, or partial scans.

Motivation & Objective

• Motivate learning a continuous latent space that captures both geometric and structural variation across diverse shape families.
• Represent shapes as hierarchy of graphs (n-ary trees with sibling relations) to preserve cross-object structural consistency.
• Enable generation, interpolation, and structure-aware editing of structured 3D shapes from partial or unannotated inputs.

Proposed method

• Propose StructureNet, a variational autoencoder with two encoders (geometry and graph) and two decoders (graph and geometry) operating on an n-ary hierarchical graph representation.
• Encode leaf parts by geometry encoders (bounding box or point cloud) and propagate through recursive graph convolutions inspired by GIN/Dynamic Graph CNNs.
• Achieve order-invariance by using symmetric functions during encoding and a linear assignment-based matching during decoding.
• Model horizontal relationships between siblings as edges (e.g., adjacency and various symmetries) to enrich structure-aware generation.
• Decode child graphs with a fixed maximum number of possible children and edges, using a two-stage decoding process with skip connections and iterative message passing.
• Train with a composite loss: reconstruction loss, structure-consistency loss, and variational regularization (beta-VAE style) to ensure a smooth latent space.

Experimental results

Research questions

• RQ1 Can a single latent space simultaneously capture geometric variations and hierarchical structure across diverse shape families?
• RQ2 How effectively can the model encode and decode n-ary hierarchies with horizontal relationships without binarization?
• RQ3 Does the structure-aware latent space enable accurate shape generation, interpolation, editing, and discovery of structure from unannotated inputs?
• RQ4 How does StructureNet perform relative to baselines on learning, generation, and structure-aware tasks across chair/PartNet datasets?

Key findings

• The model enables generation and interpolation of structured 3D shapes with both geometric and structural variations.
• StructureNet supports projecting unannotated inputs (images, point clouds) into a latent space that yields semantically segmented, hierarchical shapes.
• The encoder/decoder design with order-invariant encoding and linear assignment matching achieves robust learning on large shape families.
• Horizontal sibling relationships modeled by graph edges improve decoding coherence and enable structure-aware manipulations such as editing and part-level changes.
• Empirical evaluations show significant advantages of StructureNet over baseline and competing methods in learning structured shape distributions.