[Paper Review] Feedback Control For Cassie With Deep Reinforcement Learning

Zhaoming Xie, Glen Berseth+3 more|arXiv (Cornell University)|Mar 15, 2018

Robotic Locomotion and ControlEngineering18 references21 citations

TL;DR

This paper proposes a deep reinforcement learning (DRL) framework for training robust, model-free feedback controllers for the Cassie bipedal robot using a realistic MuJoCo simulation. By formulating imitation learning as a Markov Decision Process, the method learns controllers that track reference motions across varying speeds, demonstrating robustness to sensory delays, uneven terrain, and large external pushes, with interpolated policies enabling adaptive gait speed control.

ABSTRACT

Bipedal locomotion skills are challenging to develop. Control strategies often use local linearization of the dynamics in conjunction with reduced-order abstractions to yield tractable solutions. In these model-based control strategies, the controller is often not fully aware of many details, including torque limits, joint limits, and other non-linearities that are necessarily excluded from the control computations for simplicity. Deep reinforcement learning (DRL) offers a promising model-free approach for controlling bipedal locomotion which can more fully exploit the dynamics. However, current results in the machine learning literature are often based on ad-hoc simulation models that are not based on corresponding hardware. Thus it remains unclear how well DRL will succeed on realizable bipedal robots. In this paper, we demonstrate the effectiveness of DRL using a realistic model of Cassie, a bipedal robot. By formulating a feedback control problem as finding the optimal policy for a Markov Decision Process, we are able to learn robust walking controllers that imitate a reference motion with DRL. Controllers for different walking speeds are learned by imitating simple time-scaled versions of the original reference motion. Controller robustness is demonstrated through several challenging tests, including sensory delay, walking blindly on irregular terrain and unexpected pushes at the pelvis. We also show we can interpolate between individual policies and that robustness can be improved with an interpolated policy.

Motivation & Objective

• To develop a model-free, deep reinforcement learning-based feedback control framework for bipedal locomotion that fully exploits the dynamics of a real-world robot.
• To demonstrate that DRL can produce robust, high-performance walking controllers on a realistic simulation of the Cassie robot, overcoming limitations of model-based control.
• To enable learning of controllers for different walking speeds by re-timing the reference motion, even when physically infeasible.
• To improve controller robustness through policy interpolation between controllers trained for different speeds.
• To validate the approach under challenging conditions such as sensory delay, blind walking on irregular terrain, and external pushes.

Proposed method

• Formulate the feedback control problem as a Markov Decision Process (MDP) where the goal is to learn an optimal policy that imitates a reference motion.
• Use a multi-layer neural network to parameterize the policy, enabling end-to-end learning of control actions from state and reference trajectory inputs.
• Train the policy using policy-gradient reinforcement learning with sparse dense rewards based on tracking error and stability metrics.
• Generate reference motions for different speeds by time-scaling the original reference trajectory, even when resulting in physically infeasible motions like sliding feet.
• Implement policy interpolation by blending actions from two policies based on a parameter λ, allowing smooth transitions between gaits of different speeds.
• Use adaptive interpolation that varies λ based on the robot's current speed to naturally accelerate or decelerate on slopes.

Experimental results

Research questions

• RQ1 Can deep reinforcement learning learn robust, model-free feedback controllers for a realistic bipedal robot like Cassie without relying on simplified dynamics models?
• RQ2 How effective is DRL in learning controllers that track physically infeasible reference motions, such as time-scaled gaits with sliding feet?
• RQ3 Can interpolated policies between different speed controllers improve robustness on uneven terrain compared to single-speed policies?
• RQ4 To what extent can DRL-based controllers handle real-world challenges like sensory delays, blind walking, and large external disturbances?
• RQ5 Can the framework be extended to enable zero-shot adaptation to new reference motions, such as running or jumping?

Key findings

• The DRL-based controller successfully learned to walk at different speeds by training on time-scaled versions of the reference motion, even when the resulting motion was physically infeasible due to foot sliding.
• The controller demonstrated robustness to sensory delays of up to 100 ms, maintaining stable gait without significant performance degradation.
• On uneven terrain, the interpolated policy enabled stable walking on sinusoidal terrain with height 0.22 m, significantly outperforming non-interpolated policies.
• The controller withstood external pushes of up to 50 N forward, 70 N backward, and 15 N sideways, showing strong disturbance rejection.
• Interpolated policies allowed the robot to naturally speed up and slow down on slopes, achieving adaptive gait control without additional training.
• The framework is extendable to new behaviors via new reference motions, suggesting potential for zero-shot learning of diverse locomotion tasks.