[Paper Review] Learning Agile Robotic Locomotion Skills by Imitating Animals

Xue Bin Peng, Erwin Coumans+4 more|arXiv (Cornell University)|Apr 2, 2020

Robotic Locomotion and Control41 citations

TL;DR

The paper presents an imitation-learning framework that enables legged robots to acquire agile locomotion skills by imitating real animal motions, trained in simulation with domain randomization and transferred to the real robot via latent-space adaptation.

ABSTRACT

Reproducing the diverse and agile locomotion skills of animals has been a longstanding challenge in robotics. While manually-designed controllers have been able to emulate many complex behaviors, building such controllers involves a time-consuming and difficult development process, often requiring substantial expertise of the nuances of each skill. Reinforcement learning provides an appealing alternative for automating the manual effort involved in the development of controllers. However, designing learning objectives that elicit the desired behaviors from an agent can also require a great deal of skill-specific expertise. In this work, we present an imitation learning system that enables legged robots to learn agile locomotion skills by imitating real-world animals. We show that by leveraging reference motion data, a single learning-based approach is able to automatically synthesize controllers for a diverse repertoire behaviors for legged robots. By incorporating sample efficient domain adaptation techniques into the training process, our system is able to learn adaptive policies in simulation that can then be quickly adapted for real-world deployment. To demonstrate the effectiveness of our system, we train an 18-DoF quadruped robot to perform a variety of agile behaviors ranging from different locomotion gaits to dynamic hops and turns.

Motivation & Objective

• Motivate and formalize the challenge of achieving animal-like agility in robots through learning-based methods.
• Leverage real animal motion data to guide policy learning without hand-crafting skill-specific rewards.
• Develop a sample-efficient sim-to-real transfer via domain randomization and latent-space adaptation.
• Demonstrate learning of diverse agile behaviors on an 18-DoF quadruped and transfer to a real robot.

Proposed method

• Retarget animal motion clips to the robot morphology using inverse kinematics.
• Train a motion imitation policy in simulation to reproduce retargeted motions with a goal-conditioned input.
• Use a PD-controlled torque output and a pose/velocity based reward to match reference trajectories.
• Incorporate domain randomization to expose the policy to varied dynamics during training.
• Introduce a latent dynamics encoder that conditions the policy on a latent z representing randomized dynamics, with an information bottleneck to trade robustness and adaptability.
• Apply an advantage-weighted regression based adaptation procedure (AWR) to adapt the latent encoding z on the real robot with a small number of trials.

Experimental results

Research questions

• RQ1 Can real animal motions be effectively used to train robust, diverse locomotion skills on legged robots?
• RQ2 Does sim-to-real transfer stand up across dynamic gaits when guided by motion imitation and latent-space adaptation?
• RQ3 How does enforcing an information bottleneck on the latent dynamics encoding affect robustness and adaptability in real-world deployment?
• RQ4 What is the impact of domain randomization plus latent-space adaptation on real-world performance compared to non-adaptive or non-randomized baselines?

Key findings

• Policy adapts to a diverse set of agile skills (gaits, hops, turns) on an 18-DoF Laikago quadruped.
• Adaptive policies outperform non-adaptive baselines on most skills when transferred to the real robot.
• Adaptive methods enable dynamic skills (e.g., Dog Pace, Dog Spin) to be executed more reliably than robust but non-adaptive policies.
• Training uses ~200 million simulation samples and ~50 real-world trials per policy to adapt behaviors.
• Policies trained with latent-space adaptation generalize to a range of unseen dynamic environments better than non-adaptive policies.
• Backward-imitated mocap data (e.g., dog gait) can yield faster real-world speeds than the manufacturer gait (e.g., 1.08 m/s vs 0.84 m/s).