[Paper Review] Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis

Rudy Bunel, Matthew Hausknecht+3 more|arXiv (Cornell University)|May 11, 2018

Software Engineering ResearchComputer Science24 references67 citations

TL;DR

The paper combines reinforcement learning with supervised training and syntax-aware decoding to generate syntactically valid programs that satisfy given IO specifications, improving data efficiency and handling program aliasing in neural program synthesis.

ABSTRACT

Program synthesis is the task of automatically generating a program consistent with a specification. Recent years have seen proposal of a number of neural approaches for program synthesis, many of which adopt a sequence generation paradigm similar to neural machine translation, in which sequence-to-sequence models are trained to maximize the likelihood of known reference programs. While achieving impressive results, this strategy has two key limitations. First, it ignores Program Aliasing: the fact that many different programs may satisfy a given specification (especially with incomplete specifications such as a few input-output examples). By maximizing the likelihood of only a single reference program, it penalizes many semantically correct programs, which can adversely affect the synthesizer performance. Second, this strategy overlooks the fact that programs have a strict syntax that can be efficiently checked. To address the first limitation, we perform reinforcement learning on top of a supervised model with an objective that explicitly maximizes the likelihood of generating semantically correct programs. For addressing the second limitation, we introduce a training procedure that directly maximizes the probability of generating syntactically correct programs that fulfill the specification. We show that our contributions lead to improved accuracy of the models, especially in cases where the training data is limited.

Motivation & Objective

• Address program aliasing by optimizing for any semantically correct program rather than a single reference program.
• Leverage program syntax to prune the search space and enforce syntactic correctness.
• Enable learning of syntax when formal grammar is unavailable.
• Demonstrate the approach on the Karel DSL with limited training data to show data efficiency.

Proposed method

• Use a sequential LSTM-based model conditioned on embeddings of input-output pairs, with one decoder per IO pair and max-pooling over decoders to predict the next token.
• Represent programs as token sequences from a DSL alphabet and train with supervised maximum likelihood (MLE) as baseline.
• Introduce reinforcement learning with a reward R(lambda) for correctness against IO examples, and approximate the objective via sampling (REINFORCE).
• Improve training by beam-search-based sampling to align training with test-time decoding (BS(p_theta, S)).
• Optionally incorporate a syntax checker to prune invalid prefixes via a mask on the decoder outputs (stx conditioning).
• Propose a learned syntax checker g_phi that penalizes syntactically invalid tokens by adding its output to the decoder (joint architecture).

Experimental results

Research questions

• RQ1 Can reinforcement learning directly optimize for generating any program consistent with IO specifications rather than a single ground-truth program?
• RQ2 Does explicit syntax-aware pruning improve the generation of syntactically valid and correct programs?
• RQ3 How does jointly learning syntax affect performance when an explicit grammar is not available?
• RQ4 Is the method data-efficient, performing well with limited training data in the Karel DSL?

Key findings

• Reinforcement learning targeting correctness improves top-1 generalization versus pure supervised training, especially with larger training sets.
• Beam-search-based sampling aligned with training (RL_beam variants) yields better generalization and diversity of correct programs than standard RL or MLE.
• Diversity-promoting objectives (RL_beam_div, RL_beam_div_opt) reduce redundancy and improve relative accuracy in top predictions.
• Using a syntax checker (handwritten or learned) to prune the space yields performance gains, with learned syntax performing well when data is limited.
• RL methods show more pronounced gains under limited data, indicating improved data efficiency for program synthesis.