[Paper Review] Efficient Learning of Control Policies for Robust Quadruped Bounding using Pretrained Neural Networks

Zhicheng Wang, Anqiao Li+5 more|arXiv (Cornell University)|Nov 1, 2020

Robotic Locomotion and Control13 references4 citations

TL;DR

This paper proposes a data-efficient reinforcement learning framework that fine-tunes a pre-trained neural network policy for robust quadruped bounding gait control. By leveraging imitation from model-based controllers and optimizing with a phase- and contact-aware reward function, the method enables efficient DRL training and direct deployment on the real Jueying Mini robot, achieving stable locomotion over uneven indoor and outdoor terrains.

ABSTRACT

Bounding is one of the important gaits in quadrupedal locomotion for negotiating obstacles. The authors proposed an effective approach that can learn robust bounding gaits more efficiently despite its large variation in dynamic body movements. The authors first pretrained the neural network (NN) based on data from a robot operated by conventional model based controllers, and then further optimised the pretrained NN via deep reinforcement learning (DRL). In particular, the authors designed a reward function considering contact points and phases to enforce the gait symmetry and periodicity, which improved the bounding performance. The NN based feedback controller was learned in the simulation and directly deployed on the real quadruped robot Jueying Mini successfully. A variety of environments are presented both indoors and outdoors with the authors approach. The authors approach shows efficient computing and good locomotion results by the Jueying Mini quadrupedal robot bounding over uneven terrain.

Motivation & Objective

• To improve sample efficiency in training robust control policies for quadruped bounding gaits.
• To address the challenge of high-dimensional, dynamic movements in quadruped locomotion over uneven terrain.
• To enable direct transfer of learned policies from simulation to real-world deployment on a physical robot.
• To maintain gait symmetry and periodicity through reward shaping in reinforcement learning.
• To reduce training time and computational cost compared to end-to-end RL training.

Proposed method

• Pretrain a neural network policy using demonstrations from conventional model-based controllers on a quadruped robot.
• Fine-tune the pre-trained policy using deep reinforcement learning (DRL) in simulation.
• Design a reward function that explicitly encodes contact point timing and gait phase to enforce symmetry and periodicity.
• Use a feedback control architecture where the NN outputs joint torques based on state observations including body pose and contact states.
• Train the policy in a simulated environment with diverse terrain variations to improve robustness.
• Deploy the final policy directly on the real Jueying Mini robot without further tuning.

Experimental results

Research questions

• RQ1 Can pre-training on model-based controller demonstrations significantly reduce sample complexity in training DRL policies for quadruped bounding?
• RQ2 How effective is a phase- and contact-aware reward function in promoting symmetric and periodic gait patterns?
• RQ3 To what extent can a policy trained in simulation generalize to real-world uneven terrains?
• RQ4 What is the performance gap between end-to-end RL and pre-trained + fine-tuned RL in terms of training efficiency and robustness?
• RQ5 Can the final policy be deployed directly on hardware without domain randomization or sim-to-real transfer tricks?

Key findings

• The pre-trained and fine-tuned policy achieved stable bounding gait on the Jueying Mini robot across diverse indoor and outdoor terrains.
• The method reduced training time and sample complexity compared to end-to-end DRL training.
• The reward function with contact and phase supervision successfully enforced gait periodicity and symmetry, improving locomotion quality.
• The policy was successfully deployed on the real robot without additional sim-to-real adaptation steps.
• The system demonstrated robustness to terrain irregularities, including slopes and uneven ground.
• The approach achieved a balance between sample efficiency and real-world performance, outperforming standard DRL in training efficiency.