[Paper Review] Towards Mixed Optimization for Reinforcement Learning with Program Synthesis

Surya Bhupatiraju, Kumar Krishna Agrawal+1 more|arXiv (Cornell University)|Jul 1, 2018

Reinforcement Learning in Robotics13 references3 citations

TL;DR

This paper proposes Morl, a mixed optimization framework that iteratively improves deep reinforcement learning policies by synthesizing symbolic programs from black-box policies, repairing them for safety and performance constraints, and distilling them back into neural policies for further gradient-based optimization. The approach enables faster convergence and improved sample efficiency, demonstrated on CartPole with a 176.8 average return after fine-tuning, compared to 38.65 for unmodified policies.

ABSTRACT

Deep reinforcement learning has led to several recent breakthroughs, though the learned policies are often based on black-box neural networks. This makes them difficult to interpret and to impose desired specification constraints during learning. We present an iterative framework, MORL, for improving the learned policies using program synthesis. Concretely, we propose to use synthesis techniques to obtain a symbolic representation of the learned policy, which can then be debugged manually or automatically using program repair. After the repair step, we use behavior cloning to obtain the policy corresponding to the repaired program, which is then further improved using gradient descent. This process continues until the learned policy satisfies desired constraints. We instantiate MORL for the simple CartPole problem and show that the programmatic representation allows for high-level modifications that in turn lead to improved learning of the policies.

Motivation & Objective

• To address the limitations of black-box deep reinforcement learning policies, which are hard to interpret, debug, or constrain with safety guarantees.
• To improve policy learning by integrating symbolic program synthesis and repair into the training loop, enabling high-level modifications.
• To enhance sample efficiency and convergence speed by leveraging symbolic representations for policy refinement.
• To provide a framework that supports iterative, human- or machine-assisted repair of policies in symbolic form before retraining.
• To demonstrate the feasibility and benefits of this approach on a standard control benchmark, CartPole.

Proposed method

• Synthesize a symbolic program from a trained deep RL policy using a domain-specific language (DSL), inspired by Pirl and Viper.
• Perform program repair—either manually or automatically—on the symbolic program to satisfy desired constraints, such as safety or behavioral improvements.
• Use behavioral cloning (imitation learning) to distill the repaired symbolic program into a differentiable neural policy.
• Apply gradient-based policy optimization (e.g., TRPO) to further refine the distilled policy.
• Iterate the process: policy → program → repaired program → distilled policy → improved policy, forming a closed-loop refinement loop.
• Use a simple DSL for CartPole to instantiate the framework and validate its effectiveness through controlled experiments.

Experimental results

Research questions

• RQ1 Can symbolic program representations of deep RL policies enable more effective and interpretable policy improvement?
• RQ2 Does repairing policies in symbolic space lead to faster convergence and better sample efficiency in subsequent training?
• RQ3 Can high-level, human-provided insights encoded in program repairs significantly improve policy performance from poor initializations?
• RQ4 How does the integration of program synthesis and repair compare to standard end-to-end RL in terms of convergence speed and final performance?
• RQ5 To what extent can symbolic representations support verifiable and safe policy learning in reinforcement learning?

Key findings

• The worst-initialized policy achieved an average return of 9.28 over 25 runs, indicating poor performance from a poor starting point.
• After program repair and distillation, the intermediate policy achieved an average return of 66 after 15,000 training epochs, showing significant improvement.
• The near-optimal repaired program led to a distilled policy with an average return of 185 after 15,000 epochs, demonstrating the effectiveness of symbolic repair.
• After fine-tuning with TRPO for 25 episodes, the policy derived from the near-optimal program achieved an average return of 176.8, outperforming the baseline by over 3x.
• The TRPO training process converged significantly faster when initialized from a repaired symbolic policy, validating the hypothesis that symbolic insights accelerate learning.
• The framework successfully demonstrated that iterative repair in symbolic space enables faster convergence to near-optimal policies, especially from poor initializations.