[Paper Review] Learning to Represent Programs with Property Signatures

Augustus Odena, Charles Sutton|arXiv (Cornell University)|Feb 13, 2020

Machine Learning and Algorithms35 references18 citations

TL;DR

This paper introduces property signatures—a machine learning-friendly representation of programs and specifications based on evaluating a set of boolean-valued properties over input-output pairs. By learning to predict property signatures for composed functions, the method enables a synthesizer to find correct programs twice as fast and with significantly higher success rate than a baseline, demonstrating dramatic improvements on a diverse set of functional programming problems.

ABSTRACT

We introduce the notion of property signatures, a representation for programs and program specifications meant for consumption by machine learning algorithms. Given a function with input type $τ_{in}$ and output type $τ_{out}$, a property is a function of type: $(τ_{in}, τ_{out}) ightarrow exttt{Bool}$ that (informally) describes some simple property of the function under consideration. For instance, if $τ_{in}$ and $τ_{out}$ are both lists of the same type, one property might ask `is the input list the same length as the output list?'. If we have a list of such properties, we can evaluate them all for our function to get a list of outputs that we will call the property signature. Crucially, we can `guess' the property signature for a function given only a set of input/output pairs meant to specify that function. We discuss several potential applications of property signatures and show experimentally that they can be used to improve over a baseline synthesizer so that it emits twice as many programs in less than one-tenth of the time.

Motivation & Objective

• To develop a general-purpose representation for programs and specifications that supports machine learning.
• To improve program synthesis performance by leveraging property signatures as inductive biases.
• To enable automated discovery of useful program properties through learning, rather than manual design.
• To demonstrate that property signatures can drastically accelerate synthesis on complex, human-authored programming problems.

Proposed method

• The paper defines a property as a boolean function that checks simple invariants on input-output pairs of a target function.
• A property signature is formed by evaluating a fixed set of such properties on a function’s input-output examples, producing a binary vector.
• The method learns a composition predictor model that infers the property signature of a component function from the signature of its composition.
• The synthesizer uses property signatures to guide search, filtering candidate programs based on predicted property matches.
• The approach is evaluated in a custom, Turing-complete functional language called Searcho, designed for fast program execution during synthesis.
• The system is open-sourced, including the language, runtime, search infrastructure, and training data, to support future research in ML-guided synthesis.

Experimental results

Research questions

• RQ1 Can property signatures serve as an effective inductive bias for program synthesis?
• RQ2 Can a learned model predict property signatures of component functions from their compositions, improving synthesis efficiency?
• RQ3 Can property signatures enable synthesis of programs that a baseline synthesizer fails to find?
• RQ4 How does the performance of a property signature-augmented synthesizer compare to a baseline on complex, human-authored programming problems?

Key findings

• The property signature method synthesized over twice as many programs as the baseline synthesizer on a test set of 185 functional programming problems.
• The method achieved this performance gain in less than one-tenth of the time required by the baseline.
• More than half of the programs synthesized using property signatures were not found by the baseline even after a day of search.
• The composition predictor model successfully inferred intermediate property signatures, enabling effective premise selection during synthesis.
• The approach successfully synthesized a complex program—removing consecutive duplicates from a list—that the baseline could not find.
• The open-sourced system provides a foundation for future research in machine learning–guided program synthesis.