[Paper Review] Neuro-Symbolic Program Synthesis

Emilio Parisotto, Abdelrahman Mohamed+4 more|arXiv (Cornell University)|Nov 6, 2016

Software Testing and Debugging Techniques105 citations

TL;DR

The paper introduces Neuro-Symbolic Program Synthesis (NSPS), a framework that learns to generate DSL-based programs from input-output examples using a Recursive-Reverse-Recursive Neural Network (R3NN) conditioned on I/O encodings, demonstrated on regular-expression based string transformations.

ABSTRACT

Recent years have seen the proposal of a number of neural architectures for the problem of Program Induction. Given a set of input-output examples, these architectures are able to learn mappings that generalize to new test inputs. While achieving impressive results, these approaches have a number of important limitations: (a) they are computationally expensive and hard to train, (b) a model has to be trained for each task (program) separately, and (c) it is hard to interpret or verify the correctness of the learnt mapping (as it is defined by a neural network). In this paper, we propose a novel technique, Neuro-Symbolic Program Synthesis, to overcome the above-mentioned problems. Once trained, our approach can automatically construct computer programs in a domain-specific language that are consistent with a set of input-output examples provided at test time. Our method is based on two novel neural modules. The first module, called the cross correlation I/O network, given a set of input-output examples, produces a continuous representation of the set of I/O examples. The second module, the Recursive-Reverse-Recursive Neural Network (R3NN), given the continuous representation of the examples, synthesizes a program by incrementally expanding partial programs. We demonstrate the effectiveness of our approach by applying it to the rich and complex domain of regular expression based string transformations. Experiments show that the R3NN model is not only able to construct programs from new input-output examples, but it is also able to construct new programs for tasks that it had never observed before during training.

Motivation & Objective

• Address limitations of purely neural program induction methods (computational cost, task-specific models, and interpretability) by proposing a neuro-symbolic approach.
• Learn a generative model over DSL programs conditioned on I/O examples to enable automatic, test-time construction of consistent programs.
• Demonstrate effectiveness on complex string transformation DSLs inspired by FlashFill and evaluate generalization to unseen tasks and real-world benchmarks.

Proposed method

• Propose a cross-correlation I/O network to produce a continuous representation of a set of input-output examples.
• Introduce the Recursive-Reverse-Recursive Neural Network (R3NN) to incrementally expand partial programs into complete DSL trees.
• Encode DSL programs as parse trees with a tree-structured probabilistic generation using leaf and rule representations and global tree states.
• Condition the program generation on I/O encodings (including variants like LSTM-based, cross-correlation, and augmented cross-correlation encoders).
• Train end-to-end on a large set of DSL programs with generated I/O examples to learn to prefer expansions that match I/O specifications.

Experimental results

Research questions

• RQ1 Can NSPS synthesize DSL-based programs consistent with given input-output examples without explicit search?
• RQ2 How well does the R3NN-based generation generalize to unseen programs and unseen I/O pairs within a regex-based string transformation DSL?
• RQ3 What is the impact of different I/O encoders on the accuracy and generalization of program synthesis?
• RQ4 How does NSPS compare to a simpler io2seq approach for generating DSL trees from I/O conditioning?
• RQ5 What is the system’s performance on real-world FlashFill benchmarks in terms of solvable tasks and program size limitations?

Key findings

• NSPS can synthesize programs for new I/O examples and even for tasks not observed during training.
• A 63% success rate on unseen tasks and 94% when sampling 100 programs from the model were reported on the core benchmark.
• NSPS learns 38% of 238 real-world FlashFill benchmarks within a size-13 DSL limit.
• R3NN with appropriate I/O conditioning significantly outperforms io2seq, with io2seq achieving around 42% vs. 91% for R3NN under 100 samples.
• Pre-conditioning the I/O encodings yielded better results than post- or root-conditioning.
• Backtracking (sampling multiple programs) markedly improves accuracy, reaching near-perfect results with sufficient samples (e.g., 300 samples).
• Across FlashFill benchmarks, 10-sample results solve only about 13%, whereas broader sampling (up to 2000 samples) improves to around 38%.