[Paper Review] Sim-to-Real: Learning Agile Locomotion For Quadruped Robots

Jie Tan, Tingnan Zhang+6 more|arXiv (Cornell University)|Apr 27, 2018

Robotic Locomotion and ControlEngineering37 references113 citations

TL;DR

The paper presents a complete sim-to-real learning system for agile quadruped locomotion, learning gaits in simulation with a robust, adaptable policy deployable on real Minitaur robots, using improved fidelity, latency modeling, and dynamics/randomization to bridge the reality gap.

ABSTRACT

Designing agile locomotion for quadruped robots often requires extensive expertise and tedious manual tuning. In this paper, we present a system to automate this process by leveraging deep reinforcement learning techniques. Our system can learn quadruped locomotion from scratch using simple reward signals. In addition, users can provide an open loop reference to guide the learning process when more control over the learned gait is needed. The control policies are learned in a physics simulator and then deployed on real robots. In robotics, policies trained in simulation often do not transfer to the real world. We narrow this reality gap by improving the physics simulator and learning robust policies. We improve the simulation using system identification, developing an accurate actuator model and simulating latency. We learn robust controllers by randomizing the physical environments, adding perturbations and designing a compact observation space. We evaluate our system on two agile locomotion gaits: trotting and galloping. After learning in simulation, a quadruped robot can successfully perform both gaits in the real world.

Motivation & Objective

• Automate the design of agile quadruped locomotion policies from scratch using deep reinforcement learning.
• Narrow the reality gap between simulation and real hardware via accurate actuator models, latency handling, and system identification.
• Provide a spectrum of user controllability from fully learned to user-guided gait patterns.
• Demonstrate transfer of learned gaits (trotting and galloping) to a real Minitaur robot with improved energy efficiency.
• Evaluate how robustness techniques affect transfer performance across varied physical parameters.

Proposed method

• Formulate locomotion as a Partially Observable Markov Decision Process and optimize with Proximal Policy Optimization (PPO).
• Use a compact leg-space action representation to avoid invalid configurations and facilitate learning.
• Integrate an open-loop reference gait with a feedback policy to allow user-specified or learned gaits.
• Enhance simulation fidelity with system identification, an actuator model, and latency handling; model PWM-driven DC motors with piecewise torque-current relations.
• Employ robustness techniques including dynamics randomization, perturbations, and compact observations to improve sim-to-real transfer.
• Evaluate learned policies on trotting and galloping gaits and compare to expert-crafted gaits in real hardware.

Experimental results

Research questions

• RQ1 Can policies learned in a physics-based simulation transfer effectively to a real quadruped robot without post-training fine-tuning?
• RQ2 What combination of simulator fidelity enhancements and robustness techniques best narrows the reality gap for agile locomotion?
• RQ3 How does user-guided control (open-loop references) influence the learned gait style and balance?
• RQ4 What are the energy and speed trade-offs between learned gaits and handcrafted expert gaits on real hardware?
• RQ5 How does observation space dimensionality affect transferability and stability of the learned policies?

Key findings

• Agile galloping and trotting can emerge automatically from simulation and transfer to real Minitaur robots when the simulator is improved and robustness methods are applied.
• Learned gaits achieve competitive speeds with lower power consumption than handcrafted equivalents (galloping and trotting show substantial energy reductions).
• Accurate actuator modeling and latency-aware simulation are essential for successful real-world transfer; without them, learned policies fail on hardware.
• Dynamics randomization and random perturbations improve robustness and transferability, with a trade-off between robustness and optimality.
• A compact observation space (four-dimensional) helps transfer stability when using a reduced sensor set.
• In trotting, learned gait speed is 0.60 m/s on real robot vs 0.50 m/s in simulation, and energy use is reduced compared to handcrafted trotting (71.78 W vs 92.72 W).
• In galloping, learned gait speed is 1.18 m/s on real robot vs 1.21 m/s in simulation, with lower energy use (188.79 W vs 290.00 W).