[Paper Review] DeepCoder: Learning to Write Programs

Matej Balog, Alexander L. Gaunt+3 more|arXiv (Cornell University)|Nov 7, 2016

Software Engineering Research230 citations

TL;DR

DeepCoder trains a neural network to predict program attributes from input-output examples and uses these predictions to guide search-based program synthesis, achieving significant speedups over baselines and solving simple competition-style problems.

ABSTRACT

We develop a first line of attack for solving programming competition-style problems from input-output examples using deep learning. The approach is to train a neural network to predict properties of the program that generated the outputs from the inputs. We use the neural network's predictions to augment search techniques from the programming languages community, including enumerative search and an SMT-based solver. Empirically, we show that our approach leads to an order of magnitude speedup over the strong non-augmented baselines and a Recurrent Neural Network approach, and that we are able to solve problems of difficulty comparable to the simplest problems on programming competition websites.

Motivation & Objective

• Motivate solving programming competition–style problems from input-output examples using machine learning.
• Propose a framework (LIPS) that learns to induce program properties and guides existing solvers.
• Develop a DSL and data-generation pipeline enabling large-scale supervised learning for program synthesis.
• Demonstrate substantial speedups over non-augmented baselines and RNN approaches.

Proposed method

• Define a DSL of high-level data-manipulation functions with both simple and higher-order operations.
• Generate a large dataset of programs, their attributes, and input-output examples for training.
• Train neural networks to map sets of input-output examples to a distribution over program attributes.
• Integrate the neural predictions into search techniques (DFS, Sort and add, SMT-based Sketch and λ²) to prioritize promising parts of the program space.
• Train with negative cross-entropy loss to predict attribute presence, enabling a principled ranking of candidate programs.

Experimental results

Research questions

• RQ1 Can learning from input-output examples predict high-level program attributes that guide search over a DSL?
• RQ2 Does predicting function presence improve the efficiency of search-based program synthesis across different solvers?
• RQ3 How well does the learned guidance generalize to programs of different lengths not seen during training?
• RQ4 What is the impact of different encoder/decoder architectures on guiding program synthesis?

Key findings

• Neural guidance yields large speedups across multiple search strategies compared with baselines that rely on priors or non-augmented search.
• The Sort and add enumeration strategy benefits more from predicted attribute marginals than plain DFS, achieving the largest gains in many settings.
• DeepCoder can solve problems of difficulty comparable to the simplest programming competition problems within the DSL scope.
• The encoder–decoder model generalizes to longer and shorter programs beyond the training length, aided by the search procedure to correct for missed predictions.
• Compared to an RNN-based decoder, the proposed binary-attribute decoder with a simple feed-forward encoder trains and performs robustly.