[Paper Review] Program Synthesis Through Reinforcement Learning Guided Tree Search

Riley Simmons-Edler, Anders Miltner+1 more|arXiv (Cornell University)|Jun 8, 2018

Software Engineering Research20 references3 citations

TL;DR

This paper proposes Reinforcement Learning Guided Tree Search (RLGTS), a novel program synthesis approach that frames program generation as a Markov Decision Process solved via reinforcement learning, enhanced by a priority search tree to avoid local minima. RLGTS outperforms state-of-the-art baselines, solving at least twice as many programs as the next best method and achieving 70–100% better performance than a pure RL baseline when using the tree search component.

ABSTRACT

Program Synthesis is the task of generating a program from a provided specification. Traditionally, this has been treated as a search problem by the programming languages (PL) community and more recently as a supervised learning problem by the machine learning community. Here, we propose a third approach, representing the task of synthesizing a given program as a Markov decision process solvable via reinforcement learning(RL). From observations about the states of partial programs, we attempt to find a program that is optimal over a provided reward metric on pairs of programs and states. We instantiate this approach on a subset of the RISC-V assembly language operating on floating point numbers, and as an optimization inspired by search-based techniques from the PL community, we combine RL with a priority search tree. We evaluate this instantiation and demonstrate the effectiveness of our combined method compared to a variety of baselines, including a pure RL ablation and a state of the art Markov chain Monte Carlo search method on this task.

Motivation & Objective

• To address the limitations of existing program synthesis methods that rely on large labeled datasets or assume specific language structures.
• To combine the strengths of reinforcement learning and search-based synthesis for improved scalability and generalization across domain-specific languages.
• To reduce reliance on supervised pretraining while maintaining high success rates in synthesizing correct programs from input/output examples.
• To improve sample efficiency and robustness in program synthesis, especially when the true program length is unknown or poorly estimated.
• To enable integration with existing PL techniques by avoiding assumptions about language differentiability or training data availability.

Proposed method

• The paper models program synthesis as a Markov Decision Process (MDP), where states represent partial programs and actions are code lines, with rewards based on partial correctness.
• A deep Q-network (DQN) is trained via reinforcement learning to learn a policy that selects actions (code lines) to maximize cumulative reward toward a correct program.
• A priority search tree is integrated to guide exploration, prioritizing paths that are more likely to lead to correct solutions and reducing time spent in local minima.
• The method operates without requiring differentiable language constructs or large training datasets, relying only on executable partial programs and a reward function.
• The approach is evaluated on a subset of RISC-V assembly targeting floating-point operations, with performance measured by success rate and sample efficiency.
• The framework is designed to be composable with existing synthesis techniques from the programming languages community, enabling hybrid solutions.

Experimental results

Research questions

• RQ1 Can reinforcement learning be effectively applied to program synthesis without requiring large labeled datasets or differentiable language features?
• RQ2 How does combining RL with a priority search tree improve sample efficiency and success rate in program synthesis compared to pure RL or standard search methods?
• RQ3 How robust is the method to uncertainty in the true program length, particularly when the search depth cap exceeds the ground-truth length?
• RQ4 To what extent does the method generalize across different program lengths and action space complexities?
• RQ5 How does the performance of RLGTS compare to state-of-the-art search-based methods like MCMC, especially under varying search depth and instruction set constraints?

Key findings

• RLGTS solves at least twice as many programs as the best baseline (multi-armed bandit) across all tested program lengths.
• The integration of the priority search tree improves performance by 70–100% over the pure Q-function-only RL ablation.
• While MCMC performs competitively when the search depth is tightly bounded to the true program length, its performance degrades by 50–80% when the cap exceeds the true length by 3 lines.
• RLGTS maintains a success rate of 40–50% even when the action space is limited to 2–3 instructions, while MCMC fails to solve any programs with 4 or more instructions under 20,000 attempts.
• With a 1 million attempt limit, MCMC only matches RLGTS on 2-instruction programs and cannot solve any programs with 17 instructions, whereas RLGTS solves approximately 9% of such programs efficiently.
• Approximately 30% of the generated programs have a convex reward structure, which are trivially solvable by best-first search, but are filtered out as they are rare in human-authored programs.