[Paper Review] Interpretable Policy Specification and Synthesis through Natural Language and RL

Pradyumna Tambwekar, Andrew Silva+2 more|arXiv (Cornell University)|Jan 18, 2021

Explainable Artificial Intelligence (XAI)34 references4 citations

TL;DR

This paper proposes a novel framework that enables humans to specify robot behaviors using natural language, which is then translated into interpretable decision trees to warm-start reinforcement learning. The approach achieves 96–97% accuracy in mapping natural language to decision trees and significantly outperforms baselines (p < 0.001) by leveraging language-based policy initialization.

ABSTRACT

Policy specification is a process by which a human can initialize a robot's behaviour and, in turn, warm-start policy optimization via Reinforcement Learning (RL). While policy specification/design is inherently a collaborative process, modern methods based on Learning from Demonstration or Deep RL lack the model interpretability and accessibility to be classified as such. Current state-of-the-art methods for policy specification rely on black-box models, which are an insufficient means of collaboration for non-expert users: These models provide no means of inspecting policies learnt by the agent and are not focused on creating a usable modality for teaching robot behaviour. In this paper, we propose a novel machine learning framework that enables humans to 1) specify, through natural language, interpretable policies in the form of easy-to-understand decision trees, 2) leverage these policies to warm-start reinforcement learning and 3) outperform baselines that lack our natural language initialization mechanism. We train our approach by collecting a first-of-its-kind corpus mapping free-form natural language policy descriptions to decision tree-based policies. We show that our novel framework translates natural language to decision trees with a 96% and 97% accuracy on a held-out corpus across two domains, respectively. Finally, we validate that policies initialized with natural language commands are able to significantly outperform relevant baselines (p < 0.001) that do not benefit from our natural language-based warm-start technique.

Motivation & Objective

• To address the lack of interpretability and accessibility in current policy specification methods for robotics.
• To enable non-expert users to collaboratively define robot behaviors through natural language.
• To develop a machine learning framework that translates natural language descriptions into interpretable decision tree policies.
• To improve reinforcement learning performance by using language-specified policies as warm-starts.
• To create a first-of-its-kind corpus mapping free-form natural language to decision tree-based policies.

Proposed method

• The framework collects a new corpus of natural language descriptions paired with corresponding decision tree policies across two domains.
• It trains a neural sequence-to-tree model to map free-form natural language to structured decision trees.
• The learned policy representations are used to initialize and warm-start deep reinforcement learning agents.
• The system employs a discriminative sequence-to-tree architecture with attention mechanisms to improve alignment between language and policy structure.
• The method ensures interpretability by constraining policies to decision tree form, which are human-readable and modifiable.
• Evaluation is conducted via zero-shot transfer from language to policy, followed by RL fine-tuning on downstream tasks.

Experimental results

Research questions

• RQ1 Can natural language descriptions be accurately mapped to interpretable decision tree policies in robotic control tasks?
• RQ2 How does language-based policy initialization affect the sample efficiency and final performance of reinforcement learning agents?
• RQ3 To what extent does the proposed framework improve upon black-box policy specification baselines in terms of performance and interpretability?
• RQ4 Can non-expert users effectively specify robot behaviors using natural language with minimal supervision?
• RQ5 How generalizable is the learned language-to-policy mapping across different robotic domains?

Key findings

• The framework achieves 96% and 97% accuracy in mapping natural language to decision trees on a held-out test set across two distinct domains.
• Policies initialized using natural language commands significantly outperform baseline methods without language-based warm-start (p < 0.001).
• The use of interpretable decision trees enables human users to inspect, understand, and modify policies, enhancing collaboration.
• The corpus of language-to-policy pairs enables zero-shot transfer to new tasks with high fidelity.
• The approach demonstrates that natural language can serve as an effective and accessible modality for policy specification in robotics.