[Paper Review] Learning to Infer and Execute 3D Shape Programs

Yonglong Tian, Andrew Luo+5 more|arXiv (Cornell University)|Jan 9, 2019

3D Shape Modeling and Analysis41 references50 citations

TL;DR

The paper introduces 3D shape programs, a DSL with drawing and looping constructs, and a neural pipeline (program generator + neural program executor) that infers and executes these programs to reconstruct 3D shapes in a self-supervised manner, with guided adaptation to ShapeNet and extension to image-to-shape perception.

ABSTRACT

Human perception of 3D shapes goes beyond reconstructing them as a set of points or a composition of geometric primitives: we also effortlessly understand higher-level shape structure such as the repetition and reflective symmetry of object parts. In contrast, recent advances in 3D shape sensing focus more on low-level geometry but less on these higher-level relationships. In this paper, we propose 3D shape programs, integrating bottom-up recognition systems with top-down, symbolic program structure to capture both low-level geometry and high-level structural priors for 3D shapes. Because there are no annotations of shape programs for real shapes, we develop neural modules that not only learn to infer 3D shape programs from raw, unannotated shapes, but also to execute these programs for shape reconstruction. After initial bootstrapping, our end-to-end differentiable model learns 3D shape programs by reconstructing shapes in a self-supervised manner. Experiments demonstrate that our model accurately infers and executes 3D shape programs for highly complex shapes from various categories. It can also be integrated with an image-to-shape module to infer 3D shape programs directly from an RGB image, leading to 3D shape reconstructions that are both more accurate and more physically plausible.

Motivation & Objective

• Introduce 3D shape programs as a representation capturing both local geometry and high-level structure (symmetry, repetition).
• Develop a neural pipeline to infer shape programs from unlabeled shapes and execute them for reconstruction.
• Enable self-supervised end-to-end learning by back-propagating through a differentiable program executor.
• Demonstrate adaptation to real datasets (ShapeNet, Pix3D) via guided adaptation.
• Show extension to infer shape programs directly from RGB images for improved reconstruction and plausibility.

Proposed method

• Define a domain-specific language (DSL) for 3D shapes with Draw and For statements to model parts and regularities.
• Implement a two-tier program generator with Block LSTM and Step LSTM to predict block-level and per-block program tokens.
• Create a neural program executor (LSTM + 3D deconvolution) that renders shapes from programs and enables gradient-based training.
• Train the executor on synthetic block-program–shape pairs to enable end-to-end backpropagation for program synthesis.
• Use guided adaptation (GA) to adapt the pre-trained generator to ShapeNet by minimizing reconstruction loss via the frozen executor.
• Evaluate using IoU, Chamfer Distance (CD), and Earth Mover’s Distance (EMD), and perform generalization tests on unseen categories and Pix3D.

Experimental results

Research questions

• RQ1 Can a neural model infer interpretable 3D shape programs from unlabeled shapes?
• RQ2 Can a differentiable program executor enable end-to-end learning for 3D shape reconstruction?
• RQ3 Does guided adaptation improve generalization to real datasets like ShapeNet?
• RQ4 Can shapes be inferred and reconstructed directly from RGB images via a program-execution pipeline?

Key findings

• On synthetic data, the program generator achieves over 99.9% token accuracy and IoU 0.991 between input shapes and reconstructed shapes.
• The neural program executor attains IoU 0.93 for single drawing statements and 0.88 for compound statements.
• Shape Programs without GA achieve IoU 0.487 (table) and 0.422 (chair); with GA, IoU improves to 0.591 (table) and 0.516 (chair).
• CD improves from 0.067 (table) to 0.058 (table) and from 0.072 (chair) to 0.063 (chair) with GA; EMD improves from 0.067 to 0.058 (table) and from 0.072 to 0.060 (chair).
• Guided adaptation yields substantial stability/connectivity gains: stable percentages rise to 97.0% (table) and 96.5% (chair); stable & connected rise to 77.0% (table) and 66.0% (chair).
• Unseen-category generalization improves significantly with GA (e.g., bed: IoU 0.37 from 0.23; sofa: 0.60 from 0.30; cabinet: 0.48 from 0.25; bench: 0.42 from 0.18).