[Paper Review] Neural Sketch Learning for Conditional Program Generation

Vijayaraghavan Murali, Letao Qi+2 more|arXiv (Cornell University)|Mar 16, 2017

Software Engineering Research52 citations

TL;DR

The paper presents Bayou, a neural-symbolic system that learns over program sketches to conditionally generate type-safe Java-like code, then concretizes sketches into complete programs using combinatorial search. It demonstrates strong ability to predict API-heavy method bodies from limited labels.

ABSTRACT

We study the problem of generating source code in a strongly typed, Java-like programming language, given a label (for example a set of API calls or types) carrying a small amount of information about the code that is desired. The generated programs are expected to respect a "realistic" relationship between programs and labels, as exemplified by a corpus of labeled programs available during training. Two challenges in such conditional program generation are that the generated programs must satisfy a rich set of syntactic and semantic constraints, and that source code contains many low-level features that impede learning. We address these problems by training a neural generator not on code but on program sketches, or models of program syntax that abstract out names and operations that do not generalize across programs. During generation, we infer a posterior distribution over sketches, then concretize samples from this distribution into type-safe programs using combinatorial techniques. We implement our ideas in a system for generating API-heavy Java code, and show that it can often predict the entire body of a method given just a few API calls or data types that appear in the method.

Motivation & Objective

• Motivate conditional program generation where labels reveal limited information about target code.
• Develop a learning framework that operates on program sketches rather than raw code to manage syntactic/semantic constraints.
• Integrate neural encoding of sketches with type-guided combinatorial concretization to produce compilable programs.
• Evaluate on API-heavy Android/Java code to show generation of full method bodies from few API calls or types.

Proposed method

• Define a sketch grammar that abstracts away low-level names while preserving control flow and type information.
• Propose a Gaussian Encoder-Decoder (Ged) with a latent Z linking labels X to sketches Y via P(Y|X,θ) and a decoder P(Y|Z,θ).
• Train via maximum conditional likelihood with a variational lower bound using the re-parameterization trick.
• Use a two-stage synthesis: sample sketches from P(Y|X,θ), then concretize to full programs using a type-directed stochastic search over partially concretized sketches (PCS).
• Enforce that concretization respects type safety and API usage via a grammar for sketches and a heuristic concretization distribution P(Prog|Y).
• Evaluate Bayou on a corpus of ~150k Android methods transformed to Aml (Java-like) and compare against AST-based baselines and alternative models.

Experimental results

Research questions

• RQ1 Can conditional program generation for a Java-like language be effectively performed by learning over high-level program sketches instead of full code?
• RQ2 Does sketch-based learning plus combinatorial concretization produce type-safe, compilable programs that match target functionality given limited label information?
• RQ3 How does the Ged model compare to baseline conditional generative models and to AST-based learners in terms of accuracy and robustness under partial observability?
• RQ4 What is the practical performance (time, feasibility) of generating and ranking candidate programs from sketches?
• RQ5 Does sketch-based learning generalize well to unseen data and unseen API usage patterns?

Key findings

• Bayou can often generate complex method bodies that implement tasks not seen during training given only a few API calls or types.
• Ged-Sk (sketch-based) models outperform AST-based and full-Aml baselines, with notable advantages under partial observability.
• Ged-Sk achieves the best overall accuracy across multiple metrics and remains well-formed when concretizing sketches.
• Generation and ranking of 10 candidate programs takes about 8 seconds on average.
• Sketch-based learning provides better generalization to unseen data than direct AST/AML-based learning.