[Paper Review] Scaling data-driven robotics with reward sketching and batch reinforcement learning

Serkan Cabi, Sergio Gómez Colmenarejo+14 more|arXiv (Cornell University)|Sep 26, 2019

Reinforcement Learning in RoboticsComputer Science81 references45 citations

TL;DR

The paper presents reward sketching to learn task rewards from human preferences, retroactively labeling a large stored robot experience dataset, and then training batch RL policies from that offline data to solve diverse manipulation tasks with vision.

ABSTRACT

We present a framework for data-driven robotics that makes use of a large dataset of recorded robot experience and scales to several tasks using learned reward functions. We show how to apply this framework to accomplish three different object manipulation tasks on a real robot platform. Given demonstrations of a task together with task-agnostic recorded experience, we use a special form of human annotation as supervision to learn a reward function, which enables us to deal with real-world tasks where the reward signal cannot be acquired directly. Learned rewards are used in combination with a large dataset of experience from different tasks to learn a robot policy offline using batch RL. We show that using our approach it is possible to train agents to perform a variety of challenging manipulation tasks including stacking rigid objects and handling cloth.

Motivation & Objective

• Motivate scalable data-driven robotics by leveraging large experience datasets gathered across tasks.
• Introduce reward sketching to efficiently collect human preferences for new tasks.
• Retrospectively annotate historical data with learned task rewards to enable offline batch RL.
• Train visuomotor policies entirely from offline data using batch RL.
• Demonstrate robustness and generalization of learned policies on real robots across diverse tasks.

Proposed method

• Collect diverse robot experience including teleoperation, scripted policies, and trained agents into NeverEnding Storage (NES).
• Elicit per-timestep reward sketches from humans to learn a task-specific reward model.
• Train a neural reward model using intra-episode ranking losses to label all NES episodes for a new task.
• Apply batch reinforcement learning (distributional D4PG-like) on the labeled NES data to learn visuomotor policies from pixels.
• Deploy trained policies on real robots and iteratively refine reward models via additional annotations.
• Optionally perform online interactive insertion tasks to illustrate rapid on-line refinement.

Experimental results

Research questions

• RQ1 Can reward sketching efficiently capture task rewards for diverse manipulation tasks?
• RQ2 How well can offline (batch) RL trained on richly annotated, multi-task data transfer to real robots?
• RQ3 Does leveraging NES with task-agnostic and task-specific data improve generalization and robustness of policies?
• RQ4 What is the impact of distributional value functions in batch RL for robotics when trained from heterogeneous data?
• RQ5 How rapidly can on-line human feedback converge to effective policies for industrial tasks?

Key findings

• Policies learned entirely from offline data achieve high success on real robot tasks (e.g., 80% lift_green, 60% stack_green_on_red under normal conditions).
• Robustness to unseen objects and adversarial perturbations is demonstrated (e.g., 50%/40% success for unseen objects in lift/stack tasks).
• Distributional value functions significantly improve batch RL performance in this setting.
• Inclusion of random_watcher data (off-task exploration) substantially boosts performance, especially for harder/unseen scenarios.
• Reward sketching enables retrospective labeling of large datasets, enabling scalable learning without online robot interaction.
• In a fast, interactive USB insertion example, a policy reaches over 80% success in about 8 hours with modest on-line annotation.