[Paper Review] Program Synthesis Guided Reinforcement Learning

Yichen Yang, Jeevana Priya Inala+4 more|arXiv (Cornell University)|Feb 22, 2021

Reinforcement Learning in Robotics54 references4 citations

TL;DR

This paper proposes a program synthesis-guided reinforcement learning approach that automatically generates guiding programs for long-horizon control tasks, using model predictive program synthesis to handle partial observability by modeling uncertainty in unobserved world states. The method achieves performance comparable to an oracle with perfect program knowledge on challenging environments like a 2D Minecraft-inspired craft world and a MuJoCo Ant variant.

ABSTRACT

A key challenge for reinforcement learning is solving long-horizon planning and control problems. Recent work has proposed leveraging programs to help guide the learning algorithm in these settings. However, these approaches impose a high manual burden on the user since they must provide a guiding program for every new task they seek to achieve. We propose an approach that leverages program synthesis to automatically generate the guiding program. A key challenge is how to handle partially observable environments. We propose model predictive program synthesis, which trains a generative model to predict the unobserved portions of the world, and then synthesizes a program based on samples from this model in a way that is robust to its uncertainty. We evaluate our approach on a set of challenging benchmarks, including a 2D Minecraft-inspired ``craft'' environment where the agent must perform a complex sequence of subtasks to achieve its goal, a box-world environment that requires abstract reasoning, and a variant of the craft environment where the agent is a MuJoCo Ant. Our approach significantly outperforms several baselines, and performs essentially as well as an oracle that is given an effective program.

Motivation & Objective

• To reduce the manual burden of providing guiding programs in reinforcement learning for long-horizon planning tasks.
• To address the challenge of partial observability in complex environments where state information is incomplete.
• To develop a method that synthesizes robust programs by modeling uncertainty in unobserved world states.
• To evaluate the approach on environments requiring complex subtask sequencing and abstract reasoning.
• To demonstrate performance close to that of an oracle with access to an optimal guiding program.

Proposed method

• The approach uses model predictive program synthesis, training a generative model to predict unobserved portions of the environment state.
• It samples from the generative model to synthesize programs that are robust to uncertainty in the predicted states.
• The method integrates program synthesis with reinforcement learning by using synthesized programs as inductive biases to guide exploration and policy learning.
• It employs a planning mechanism that accounts for uncertainty in state predictions, improving generalization in partially observable settings.
• The framework is trained end-to-end, with the generative model and program synthesizer jointly optimized to improve task performance.
• The approach is evaluated using a combination of imitation and reinforcement learning signals to guide program generation and policy training.

Experimental results

Research questions

• RQ1 Can program synthesis be used to automatically generate guiding programs for reinforcement learning without manual specification?
• RQ2 How can uncertainty in partially observable environments be effectively modeled and leveraged during program synthesis?
• RQ3 To what extent can an automatically synthesized program achieve performance comparable to an oracle-provided program?
• RQ4 How does the method scale across environments with varying degrees of observation sparsity and task complexity?
• RQ5 What is the impact of uncertainty-aware program synthesis on long-horizon task success rates?

Key findings

• The proposed method significantly outperforms multiple strong baselines on the 2D Minecraft-inspired craft environment, achieving higher sample efficiency and task success rates.
• In the box-world environment requiring abstract reasoning, the method demonstrated robust performance, indicating effective handling of complex, multi-step reasoning tasks.
• On the MuJoCo Ant variant of the craft environment, the approach achieved performance nearly matching that of an oracle with an optimal guiding program.
• The model predictive program synthesis component effectively reduced the impact of state uncertainty, leading to more reliable policy learning in partially observable settings.
• The method showed strong generalization across diverse environments, suggesting scalability beyond the specific benchmarks tested.