[Paper Review] Reinforcement and Imitation Learning for Diverse Visuomotor Skills

Yuke Zhu, Ziyu Wang+9 more|arXiv (Cornell University)|Feb 26, 2018

Reinforcement Learning in RoboticsComputer Science47 references116 citations

TL;DR

A model-free deep RL method that combines demonstrations with reinforcement learning to learn end-to-end visuomotor policies from pixels for diverse robotic manipulation tasks, with promising sim2real transfer.

ABSTRACT

We propose a model-free deep reinforcement learning method that leverages a small amount of demonstration data to assist a reinforcement learning agent. We apply this approach to robotic manipulation tasks and train end-to-end visuomotor policies that map directly from RGB camera inputs to joint velocities. We demonstrate that our approach can solve a wide variety of visuomotor tasks, for which engineering a scripted controller would be laborious. In experiments, our reinforcement and imitation agent achieves significantly better performances than agents trained with reinforcement learning or imitation learning alone. We also illustrate that these policies, trained with large visual and dynamics variations, can achieve preliminary successes in zero-shot sim2real transfer. A brief visual description of this work can be viewed in https://youtu.be/EDl8SQUNjj0

Motivation & Objective

• Reduce exploration difficulty in continuous visuomotor control by leveraging demonstrations.
• Integrate imitation learning with reinforcement learning in a unified training framework.
• Exploit privileged simulation information to stabilize and accelerate learning.
• Diversify training conditions to improve generalization and sim2real transfer.
• Demonstrate zero-shot transfer potential to real robots with domain randomization.

Proposed method

• Use a hybrid reward combining imitation reward from Generative Adversarial Imitation Learning (GAIL) with a task reward.
• Train an end-to-end visuomotor policy that maps RGB observations and proprioceptive features to 9-DoF joint velocity commands.
• Leverage privileged simulation states by (i) curriculum from demonstration states, (ii) separate-value learning from physical states, (iii) object-centric GAIL discriminator with masking, (iv) auxiliary state-prediction tasks.
• Represent policy with CNN for pixels, MLP for proprioception, and an LSTM core for temporal integration.
• Train in MuJoCo with diverse, procedurally generated objects and randomized dynamics; transfer to real robot via sim2real domain randomization.
• Allow demonstrations to be collected with different bodies since the discriminator uses state trajectories rather than demonstrator actions.

Experimental results

Research questions

• RQ1 Can a hybrid reinforcement learning and imitation learning framework solve long-horizon visuomotor tasks from pixel inputs where pure RL or pure IL fail?
• RQ2 Does leveraging privileged simulation data and curriculum strategies stabilize learning and accelerate convergence?
• RQ3 What is the impact of object-centric discrimination, separate-state value learning, and auxiliary tasks on policy performance?
• RQ4 To what extent can policies trained in simulation transfer to real hardware without fine-tuning (zero-shot sim2real)?

Key findings

• The full hybrid model solves all six manipulation tasks, outperforming pure RL or pure GAIL baselines.
• Demonstration-driven curriculum starting states significantly speeds up learning compared with learning from random starts.
• Learning the value function from low-dimensional physical states stabilizes training.
• Object-centric discriminators improve GAIL signaling by focusing on task-relevant features.
• Zero-shot transfer to a real Kinova Jaco arm achieved 64% success on lifting and 35% on stacking in fixed configurations, with reaching always successful in tested trajectories.
• Hybrid RL/IL with domain randomization yields robustness to large visual and dynamic variations.