[Paper Review] Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning

Abhishek Gupta, Vikash Kumar+3 more|arXiv (Cornell University)|Oct 25, 2019

Reinforcement Learning in RoboticsComputer Science116 citations

TL;DR

Relay Policy Learning (RPL) combines imitation from unstructured demos with hierarchical reinforcement learning to solve long-horizon robotic tasks, followed by fine-tuning through RL. It uses relay data relabeling to train bi-level, goal-conditioned policies and achieves improved performance over baselines.

ABSTRACT

We present relay policy learning, a method for imitation and reinforcement learning that can solve multi-stage, long-horizon robotic tasks. This general and universally-applicable, two-phase approach consists of an imitation learning stage that produces goal-conditioned hierarchical policies, and a reinforcement learning phase that finetunes these policies for task performance. Our method, while not necessarily perfect at imitation learning, is very amenable to further improvement via environment interaction, allowing it to scale to challenging long-horizon tasks. We simplify the long-horizon policy learning problem by using a novel data-relabeling algorithm for learning goal-conditioned hierarchical policies, where the low-level only acts for a fixed number of steps, regardless of the goal achieved. While we rely on demonstration data to bootstrap policy learning, we do not assume access to demonstrations of every specific tasks that is being solved, and instead leverage unstructured and unsegmented demonstrations of semantically meaningful behaviors that are not only less burdensome to provide, but also can greatly facilitate further improvement using reinforcement learning. We demonstrate the effectiveness of our method on a number of multi-stage, long-horizon manipulation tasks in a challenging kitchen simulation environment. Videos are available at https://relay-policy-learning.github.io/

Motivation & Objective

• Motivate solving multi-stage, long-horizon robotic tasks with minimal manual task labeling.
• Bootstrap hierarchical policies from unstructured demonstrations to facilitate later RL fine-tuning.
• Introduce relay data relabeling to create goal-conditioned datasets for both high- and low-level policies.
• Enable RL fine-tuning that preserves a simple, goal-conditioned reward structure and improves sample efficiency.

Proposed method

• Propose a bi-level hierarchical policy with a high-level goal setter and a low-level subgoal-conditioned policy.
• Use fixed high-level planning intervals (H) so the high-level sets subgoals for H steps while the low-level acts at every step.
• Introduce relay data relabeling to generate goal-conditioned datasets from unstructured demonstrations for both levels (Algorithms 2 and 3).
• Train high- and low-level policies via supervised imitation learning on relabeled data to initialize policies (relay imitation learning, RIL).
• Fine-tune policies with goal-conditioned natural policy gradient (NPG) while incorporating demonstrations through a max-likelihood term to leverage relabeled data (relay reinforcement fine-tuning, RRF).
• Distill multiple fine-tuned behaviors into a single multi-task policy for generalization.

Experimental results

Research questions

• RQ1 Can unstructured, unsegmented demonstrations bootstrap effective hierarchical policies via imitation learning?
• RQ2 Are relay-imitation-learned policies more amenable to reinforcement fine-tuning than flat or scratch-learned policies?
• RQ3 Can relay policy learning solve complex long-horizon manipulation tasks in a kitchen-like environment?
• RQ4 Does distilling multiple fine-tuned tasks into a single multi-task policy retain performance across goals?

Key findings

• RIL improves imitation learning over flat goal-conditioned imitation, even with unlabelled demonstrations.
• RL fine-tuning of relay policies significantly outperforms baselines, with notable gains from incorporating demonstrations during fine-tuning (RRF).
• A distillation step yields a single multi-task policy capable of solving multiple compound goals.
• Window size and reward design critically affect performance; larger windows degrade imitation and fine-tuning, and sparse rewards work best when exploration is directed.
• RPL outperforms hierarchical RL from scratch and flat imitation-learning baselines on long-horizon kitchen tasks.