[Paper Review] Learning to Infer Shape Programs Using Latent Execution Self Training.

Homer Walke, R. Kenny Jones+1 more|arXiv (Cornell University)|Nov 25, 2020

Manufacturing Process and Optimization14 references4 citations

TL;DR

This paper proposes Latent Execution Self Training (LEST), a method for inferring shape programs by self-training on unlabeled shapes using executable latent programs. By executing predicted programs to generate synthetic shapes and fine-tuning on these, LEST avoids local minima and achieves faster, more accurate reconstruction than reinforcement learning in constructive solid geometry and assembly modeling.

ABSTRACT

Inferring programs which generate 2D and 3D shapes is important for reverse engineering, enabling shape editing, and more. Supervised learning is hard to apply to this problem, as paired (program, shape) data rarely exists. Recent approaches use supervised pre-training with randomly-generated programs and then refine using self-supervised learning. But self-supervised learning either requires that the program execution process be differentiable or relies on reinforcement learning, which is unstable and slow to converge. In this paper, we present a new approach for learning to infer shape programs, which we call latent execution self training (LEST). As with recent prior work, LEST starts by training on randomly-generated (program, shape) pairs. As its name implies, it is based on the idea of self-training: running a model on unlabeled input shapes, treating the predicted programs as ground truth latent labels, and training again. Self-training is known to be susceptible to local minima. LEST circumvents this problem by leveraging the fact that predicted latent programs are executable: for a given shape $\mathbf{x}^* \in S^*$ and its predicted program $\mathbf{z} \in P$, we execute $\mathbf{z}$ to obtain a shape $\mathbf{x} \in S$ and train on $(\mathbf{z} \in P, \mathbf{x} \in S)$ pairs, rather than $(\mathbf{z} \in P, \mathbf{x}^* \in S^*)$ pairs. Experiments show that the distribution of executed shapes $S$ converges toward the distribution of real shapes $S^*$. We establish connections between LEST and algorithms for learning generative models, including variational Bayes, wake sleep, and expectation maximization. For constructive solid geometry and assembly-based modeling, LEST's inferred programs converge to high reconstruction accuracy significantly faster than those of reinforcement learning.

Motivation & Objective

• Address the challenge of inferring shape programs without paired (program, shape) training data, which is scarce in practice.
• Overcome limitations of prior self-supervised methods that rely on differentiable execution or unstable reinforcement learning.
• Improve convergence and accuracy in shape program inference by leveraging executable latent programs as pseudo-labels.
• Enable efficient learning in 2D and 3D shape generation tasks such as constructive solid geometry and assembly modeling.
• Establish theoretical connections to variational inference and expectation maximization for better training dynamics.

Proposed method

• Train an initial model on randomly generated (program, shape) pairs to establish a base prior.
• Use the model to predict programs for unlabeled input shapes, treating these as latent labels.
• Execute the predicted programs to generate synthetic shapes, forming new (program, shape) training pairs.
• Fine-tune the model on the executed shape pairs instead of the original input shapes, enabling stable self-training.
• Leverage the executability of latent programs to ensure that the distribution of generated shapes converges toward real shape distributions.
• Connect the training process to established algorithms like variational Bayes, wake-sleep, and expectation maximization for theoretical grounding.

Experimental results

Research questions

• RQ1 Can self-training for shape program inference be made more stable and effective by using executable latent programs as pseudo-labels?
• RQ2 How does LEST compare to reinforcement learning in terms of convergence speed and reconstruction accuracy for shape generation?
• RQ3 To what extent does the distribution of executed shapes converge toward the distribution of real shapes during training?
• RQ4 What theoretical connections exist between LEST and established generative model training algorithms like EM and wake-sleep?
• RQ5 Can LEST generalize across different shape modeling paradigms, such as constructive solid geometry and assembly-based modeling?

Key findings

• LEST achieves significantly faster convergence to high reconstruction accuracy compared to reinforcement learning baselines in both constructive solid geometry and assembly-based modeling.
• The distribution of shapes generated by executing latent programs converges toward the distribution of real shapes, indicating effective distributional alignment.
• Self-training with executable latent programs mitigates the risk of local minima, improving training stability over standard self-training approaches.
• LEST demonstrates strong performance without requiring differentiable execution or reward shaping, unlike prior self-supervised methods.
• The method establishes theoretical links to variational inference and expectation maximization, providing a principled foundation for its training dynamics.
• LEST outperforms reinforcement learning in both speed and accuracy, particularly in complex shape generation tasks with sparse supervision.