[Paper Review] ShapeCoder: Discovering Abstractions for Visual Programs from Unstructured Primitives

R. Kenny Jones, Paul Guerrero+2 more|arXiv (Cornell University)|May 9, 2023

Software Engineering Research4 citations

TL;DR

ShapeCoder is a novel system that discovers high-level abstraction functions and compact visual programs from unstructured primitive-based shape datasets, using a shape-to-program recognition network and e-graphs with conditional rewrites. It achieves superior compression and abstraction quality on complex 3D shape datasets without requiring annotated programs, hierarchical part labels, or program order constraints.

ABSTRACT

Programs are an increasingly popular representation for visual data, exposing compact, interpretable structure that supports manipulation. Visual programs are usually written in domain-specific languages (DSLs). Finding "good" programs, that only expose meaningful degrees of freedom, requires access to a DSL with a "good" library of functions, both of which are typically authored by domain experts. We present ShapeCoder, the first system capable of taking a dataset of shapes, represented with unstructured primitives, and jointly discovering (i) useful abstraction functions and (ii) programs that use these abstractions to explain the input shapes. The discovered abstractions capture common patterns (both structural and parametric) across the dataset, so that programs rewritten with these abstractions are more compact, and expose fewer degrees of freedom. ShapeCoder improves upon previous abstraction discovery methods, finding better abstractions, for more complex inputs, under less stringent input assumptions. This is principally made possible by two methodological advancements: (a) a shape to program recognition network that learns to solve sub-problems and (b) the use of e-graphs, augmented with a conditional rewrite scheme, to determine when abstractions with complex parametric expressions can be applied, in a tractable manner. We evaluate ShapeCoder on multiple datasets of 3D shapes, where primitive decompositions are either parsed from manual annotations or produced by an unsupervised cuboid abstraction method. In all domains, ShapeCoder discovers a library of abstractions that capture high-level relationships, remove extraneous degrees of freedom, and achieve better dataset compression compared with alternative approaches. Finally, we investigate how programs rewritten to use discovered abstractions prove useful for downstream tasks.

Motivation & Objective

• To address the challenge of discovering meaningful, reusable abstraction functions for visual programs from raw, unstructured primitive decompositions of shapes.
• To overcome limitations of prior abstraction discovery methods that require strong input assumptions, such as curated example sequences, hierarchical annotations, or predefined program orderings.
• To improve program compactness and interpretability by learning abstractions that capture both structural and parametric patterns across shape datasets.
• To enable effective downstream use of discovered abstractions in tasks like shape editing, generation, and analysis through a unified, end-to-end learning framework.

Proposed method

• Employs an iterative dream-wake-proposal-integration loop inspired by DreamCoder, enabling progressive library learning for visual programs.
• Uses a bottom-up shape-to-program recognition network that solves sub-problems and stitches solutions via combinator operations, allowing inference even on complex or unstructured inputs.
• Leverages e-graphs augmented with conditional rewrite rules to efficiently explore and apply abstractions with complex parametric expressions in a tractable manner.
• Introduces a conditional rewrite scheme that avoids unnecessary node expansion, reducing e-graph blowup and enabling exploration of richer program structures under computational constraints.
• Operates directly on primitive collections without requiring ground-truth programs, part hierarchies, or curriculum-based training data.
• Uses a compression-based objective to guide the discovery of abstractions that minimize program length while preserving shape fidelity.

Experimental results

Research questions

• RQ1 Can a system discover meaningful, reusable abstraction functions for visual programs directly from unstructured primitive decompositions of shapes, without annotated programs or part hierarchies?
• RQ2 How can e-graphs be enhanced with conditional rewrites to efficiently explore complex parametric abstractions in visual program synthesis?
• RQ3 To what extent can a neural recognition network trained in a bottom-up fashion infer complete visual programs from primitive inputs, even when full programs are not directly observable?
• RQ4 How does ShapeCoder’s abstraction discovery compare to state-of-the-art methods in terms of program compression, generalization, and robustness to noisy primitive inputs?

Key findings

• ShapeCoder discovers a library of abstractions that achieve significantly better dataset compression than baseline methods, even on complex 3D shape datasets with no additional supervision.
• The system successfully generalizes to shapes not seen during training, producing compact, interpretable programs that use discovered abstractions.
• ShapeCoder outperforms prior methods in discovering abstractions for datasets with high structural and parametric complexity, especially under relaxed input assumptions.
• The method remains effective even when primitive decompositions are noisy or generated by an unsupervised cuboid abstraction method, demonstrating robustness to input quality.
• The use of conditional rewrites in e-graphs prevents exponential blowup and enables exploration of richer program structures than previous approaches.
• Despite limitations in e-graph saturation, ShapeCoder discovers high-level, semantically meaningful abstractions that suppress spurious degrees of freedom in visual programs.