[Paper Review] SCORES: Shape Composition with Recursive Substructure Priors

Chenyang Zhu, Kai Xu+3 more|arXiv (Cornell University)|Sep 14, 2018

3D Shape Modeling and Analysis27 references16 citations

TL;DR

SCORES introduces a recursive neural network that learns hierarchical substructure priors to compose novel, coherent 3D shapes from incompatible parts by reconfiguring, adding, or removing components. It uses a recursive autoencoder with per-node plausibility losses to optimize part structures, enabling high-quality, structurally plausible compositions beyond training exemplars.

ABSTRACT

We introduce SCORES, a recursive neural network for shape composition. Our network takes as input sets of parts from two or more source 3D shapes and a rough initial placement of the parts. It outputs an optimized part structure for the composed shape, leading to high-quality geometry construction. A unique feature of our composition network is that it is not merely learning how to connect parts. Our goal is to produce a coherent and plausible 3D shape, despite large incompatibilities among the input parts. The network may significantly alter the geometry and structure of the input parts and synthesize a novel shape structure based on the inputs, while adding or removing parts to minimize a structure plausibility loss. We design SCORES as a recursive autoencoder network. During encoding, the input parts are recursively grouped to generate a root code. During synthesis, the root code is decoded, recursively, to produce a new, coherent part assembly. Assembled shape structures may be novel, with little global resemblance to training exemplars, yet have plausible substructures. SCORES therefore learns a hierarchical substructure shape prior based on per-node losses. It is trained on structured shapes from ShapeNet, and is applied iteratively to reduce the plausibility loss.We showresults of shape composition from multiple sources over different categories of man-made shapes and compare with state-of-the-art alternatives, demonstrating that our network can significantly expand the range of composable shapes for assembly-based modeling.

Motivation & Objective

• To address the challenge of composing structurally plausible 3D shapes from incompatible parts that cannot be simply connected.
• To enable shape composition that goes beyond mere part connection by allowing significant geometric and structural reconfiguration.
• To learn a hierarchical substructure prior from ShapeNet data that supports novel, coherent shape synthesis.
• To minimize plausibility loss through iterative refinement of part structures using a recursive autoencoder architecture.
• To operate without part semantics or relational information, relying only on oriented bounding boxes (OBBs) and spatial relationships.

Proposed method

• SCORES employs a recursive autoencoder: parts are encoded hierarchically into a root code via bottom-up aggregation of substructures.
• During decoding, the root code is recursively expanded into a new, coherent part assembly, enabling structure synthesis.
• Per-node plausibility losses are used to guide the optimization, ensuring substructures are geometrically and topologically plausible at all levels.
• The network is trained on segmented 3D shapes from ShapeNet, using a VQ-VAE-like codebook to discretize latent representations.
• Structure adjustment occurs iteratively through two passes: bottom-up encoding and top-down decoding with code refinement.
• The method allows adding or removing parts during composition to resolve incompatibilities and minimize plausibility loss.

Experimental results

Research questions

• RQ1 Can a neural network learn a hierarchical substructure prior that enables plausible composition of 3D shapes from incompatible parts?
• RQ2 To what extent can the network synthesize novel, coherent structures not seen in training data?
• RQ3 How effective is the recursive autoencoder with per-node plausibility losses in resolving large geometric and topological incompatibilities?
• RQ4 Can the method operate without part semantics or explicit relational information, relying only on OBBs and spatial context?
• RQ5 What are the failure modes when substructures are rare or cross-category parts are merged?

Key findings

• SCORES successfully composes novel 3D shapes with little global resemblance to training exemplars, yet maintains plausible substructures at all levels.
• The method significantly expands the range of composable shapes compared to state-of-the-art alternatives, especially for creative, non-traditional compositions.
• Failure cases occur when substructures are rare in training data, causing the network to replace them with more common but incorrect configurations.
• Cross-category merging (e.g., chair and table parts) leads to implausible results when the latent space is biased toward one category.
• The network can add or remove parts during composition, such as inserting a bicycle seat or removing a redundant chair back, to improve plausibility.
• The approach is robust to unlabeled OBBs and does not require part semantics, relying instead on spatial context and learned substructure priors.