[Paper Review] Program Synthesis and Semantic Parsing with Learned Code Idioms

Richard Shin, Miltiadis Allamanis+2 more|arXiv (Cornell University)|Jun 26, 2019

Software Engineering Research41 references44 citations

TL;DR

Patois interleaves high-level code idioms with low-level tokens by mining common AST fragments from a corpus and training a tree-based neural synthesizer to emit idioms during code generation, improving semantic parsing accuracy on Hearthstone and Spider datasets.

ABSTRACT

Program synthesis of general-purpose source code from natural language specifications is challenging due to the need to reason about high-level patterns in the target program and low-level implementation details at the same time. In this work, we present PATOIS, a system that allows a neural program synthesizer to explicitly interleave high-level and low-level reasoning at every generation step. It accomplishes this by automatically mining common code idioms from a given corpus, incorporating them into the underlying language for neural synthesis, and training a tree-based neural synthesizer to use these idioms during code generation. We evaluate PATOIS on two complex semantic parsing datasets and show that using learned code idioms improves the synthesizer's accuracy.

Motivation & Objective

• Motivate the challenge of synthesizing real-world programs in general-purpose languages by bridging high-level reasoning and low-level implementation details.
• Propose automatic mining of common code idioms from a corpus to extend the grammar with idioms as first-class operators.
• Train a tree-based neural synthesizer that can emit idioms or their bodies to interleave reasoning levels during generation.
• Demonstrate that idiom-aware training improves synthesis accuracy on semantic parsing benchmarks.
• Showcase how idioms compress and structure program fragments to ease generation and generalize across datasets.

Proposed method

• Mine frequent code idioms from a corpus using a nonparametric Bayesian approach (Pitman-Yor process) to build a posterior pTSG of idioms.
• Extend the underlying grammar with idiom fragments and label their nonterminal leaves to act as named arguments.
• Train a tree- or graph-based neural generator to emit either idioms or original AST fragments during program construction.
• Use a specialized training objective that encourages emitting idioms when beneficial, while preserving the original generation trace.
• At inference, replace idiom emissions with their body to unroll the IDE as if it were expanded step-by-step.
• Evaluate on Hearthstone (Python programs) and Spider (SQL) semantic parsing tasks using exact-match and BLEU metrics.

Experimental results

Research questions

• RQ1 Can automatically mined code idioms improve neural program synthesis when generating from natural language descriptions in general-purpose languages?
• RQ2 Does interleaving high-level idioms with low-level tokens at every generation step yield better semantic parsing accuracy than end-to-end token-level generation?
• RQ3 How does the size and quality of the idiom set (K) affect synthesis performance on different domains?
• RQ4 Can a tree-based decoder effectively utilize idioms via a training objective that preserves the full idiom bodies during inference?
• RQ5 Do idioms generalize across datasets with different schemas and programming tasks (Python vs SQL)?

Key findings

• Idioms mined from training data improve BLEU scores and can boost exact-match accuracy, notably with larger idiom sets (e.g., K=80) on Hearthstone test data.
• On Hearthstone, the best idiom configurations yield substantial gains in exact match (e.g., up to 4.5% improvement) and BLEU scores.
• On Spider, idiom-enabled models achieve or exceed baseline exact-match performance across several K values.
• The approach demonstrates that learned idioms capture frequent, semantically meaningful patterns such as common syntactic constructs and domain APIs.
• Idioms can be large fragments, enabling single-step emission of substantial code blocks and easing the decoder’s task.