[Paper Review] COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning

Avi Singh, Albert S. Yu+4 more|arXiv (Cornell University)|Oct 27, 2020

Robot Manipulation and Learning42 references38 citations

TL;DR

COG uses offline reinforcement learning to fuse task-specific data with large unlabeled prior datasets, enabling a policy to compose previously learned behaviors to solve new multi-step tasks from novel initial conditions.

ABSTRACT

Reinforcement learning has been applied to a wide variety of robotics problems, but most of such applications involve collecting data from scratch for each new task. Since the amount of robot data we can collect for any single task is limited by time and cost considerations, the learned behavior is typically narrow: the policy can only execute the task in a handful of scenarios that it was trained on. What if there was a way to incorporate a large amount of prior data, either from previously solved tasks or from unsupervised or undirected environment interaction, to extend and generalize learned behaviors? While most prior work on extending robotic skills using pre-collected data focuses on building explicit hierarchies or skill decompositions, we show in this paper that we can reuse prior data to extend new skills simply through dynamic programming. We show that even when the prior data does not actually succeed at solving the new task, it can still be utilized for learning a better policy, by providing the agent with a broader understanding of the mechanics of its environment. We demonstrate the effectiveness of our approach by chaining together several behaviors seen in prior datasets for solving a new task, with our hardest experimental setting involving composing four robotic skills in a row: picking, placing, drawer opening, and grasping, where a +1/0 sparse reward is provided only on task completion. We train our policies in an end-to-end fashion, mapping high-dimensional image observations to low-level robot control commands, and present results in both simulated and real world domains. Additional materials and source code can be found on our project website: https://sites.google.com/view/cog-rl

Motivation & Objective

• Motivate how prior, task-agnostic data can extend policy generalization in robotics.
• Propose a simple, data-driven method to stitch together behaviors via offline reinforcement learning without explicit hierarchies.
• Demonstrate that prior data can aid learning of new, multi-stage tasks from unseen initial conditions.
• Showend-to-end learning from visual observations to low-level control using offline data and sparse rewards.

Proposed method

• Extend conservative Q-learning (CQL) to incorporate both prior and task-specific data in offline RL.
• Initialize the replay buffer with prior data labeled with zero rewards, then train on a mix of prior and task data.
• Use Q-learning dynamics to propagate value from the task-rewarded trajectories into regions covered by prior data.
• Optionally fine-tune the offline policy with limited online interaction after offline training.
• Train end-to-end networks (ConvNets) to map 48x48 or 64x64 images and robot state to continuous 6-DoF actions plus discrete gripper controls.

Experimental results

Research questions

• RQ1 Can model-free offline RL leverage task-agnostic prior datasets to learn new skills?
• RQ2 Can policies solve new tasks from novel initial conditions by stitching together behaviors seen in prior data?
• RQ3 How does offline RL with prior data compare to behavioral cloning baselines for incorporating prior data?
• RQ4 Is online fine-tuning necessary or beneficial after offline learning with prior data?
• RQ5 To what extent can the approach generalize to real-world robotic setups beyond simulation?

Key findings

• COG enables solving multi-stage tasks by composing drawer opening, grasping, and obstruction removal despite never seeing complete sequences in the data.
• COG outperforms behavioral cloning baselines, SAC, and ablations across novel initial conditions in simulation.
• Online fine-tuning further raises success rates to over 90% for drawer tasks with relatively modest additional data.
• In real-world experiments, the method achieved 7/8 successes when the drawer started closed, outperforming the BC-oracle baseline.
• BC-init fails to solve unseen initial conditions, highlighting the value of offline data integration over pre-training alone.