[Paper Review] Programmatically Interpretable Reinforcement Learning

Abhinav Verma, Vijayaraghavan Murali+3 more|arXiv (Cornell University)|Apr 6, 2018

Adversarial Robustness in Machine Learning35 references97 citations

TL;DR

Pirl learns policies expressed as human-readable programs, guided by a neural policy oracle (Ndps), achieving interpretable and verifiable control policies that transfer better in some cases than deep RL.

ABSTRACT

We present a reinforcement learning framework, called Programmatically Interpretable Reinforcement Learning (PIRL), that is designed to generate interpretable and verifiable agent policies. Unlike the popular Deep Reinforcement Learning (DRL) paradigm, which represents policies by neural networks, PIRL represents policies using a high-level, domain-specific programming language. Such programmatic policies have the benefits of being more easily interpreted than neural networks, and being amenable to verification by symbolic methods. We propose a new method, called Neurally Directed Program Search (NDPS), for solving the challenging nonsmooth optimization problem of finding a programmatic policy with maximal reward. NDPS works by first learning a neural policy network using DRL, and then performing a local search over programmatic policies that seeks to minimize a distance from this neural "oracle". We evaluate NDPS on the task of learning to drive a simulated car in the TORCS car-racing environment. We demonstrate that NDPS is able to discover human-readable policies that pass some significant performance bars. We also show that PIRL policies can have smoother trajectories, and can be more easily transferred to environments not encountered during training, than corresponding policies discovered by DRL.

Motivation & Objective

• Motivate the need for interpretable and verifiable policies in reinforcement learning, especially for safety-critical applications.
• Propose a high-level programming language for policies to enable interpretability and verification.
• Introduce Neurally Directed Program Synthesis (Ndps) to search for optimal programmatic policies by imitating a neural oracle.
• Demonstrate that Ndps can discover human-readable policies in Torcs car racing and classic control tasks.
• Show that programmatic policies can be smoother, more transferable, and amenable to symbolic verification.

Proposed method

• Define a domain-specific, side-effect-free functional policy language with atoms and sequences to describe observations and actions.
• Use sketches to constrain the policy space, encoding prior structure and enabling pruning.
• Introduce Ndps, which first trains a neural policy (oracle) and then performs a local search over programmatic policies to minimize distance to the oracle on a set of interesting inputs.
• Employ input augmentation to enrich the history set H with trajectories from the current policy.
• Optimize over program templates with parameters using Bayesian optimization and, in some cases, SMT-based methods.
• Evaluate on Torcs car racing (Practice Mode) with PID-like program sketches and on three classic control games.

Experimental results

Research questions

• RQ1 Can a high-level policy language and sketch-based search yield interpretable yet performant policies in reinforcement learning tasks?
• RQ2 Does Ndps produce human-readable programs that approximate neural policies while retaining certain performance bars?
• RQ3 Are programmatic policies smoother, more robust to partial observability, and more transferable to unseen environments than deep RL policies?
• RQ4 Can symbolic verification techniques be applied to Ndps-generated policies to guarantee certain properties?
• RQ5 How does transferability of Ndps policies compare to DRL across new tracks/environments?

Key findings

• Ndps discovers interpretable, human-readable policies that pass significant performance bars, though not always matching DRL performance.
• Ndps policies exhibit smoother trajectories and can tolerate more noise than corresponding neural policies.
• Ndps policies demonstrate robustness to missing/noisy features and show superior transfer to unseen tracks compared to DRL.
• Programmatic policies lend themselves to verification using standard symbolic techniques, enabling formal reasoning about behavior.
• In Torcs, Ndps policies can complete laps and generalize to new tracks better than the DRL baseline in transfer scenarios.