[Paper Review] Learning multiple gaits of quadruped robot using hierarchical reinforcement learning

Yun-Ho Kim, Bukun Son+1 more|arXiv (Cornell University)|Dec 9, 2021

Robotic Locomotion and Control4 citations

TL;DR

This paper proposes a hierarchical reinforcement learning controller for quadruped robots that learns multiple gaits (trot, pace, bound) based on velocity commands, using a two-level policy: a central pattern generator (CPG) for gait selection and a local feedback controller for joint tracking. The method achieves energy-efficient locomotion across velocity ranges, outperforming single-policy approaches that default to one gait regardless of speed.

ABSTRACT

There is a growing interest in learning a velocity command tracking controller of quadruped robot using reinforcement learning due to its robustness and scalability. However, a single policy, trained end-to-end, usually shows a single gait regardless of the command velocity. This could be a suboptimal solution considering the existence of optimal gait according to the velocity for quadruped animals. In this work, we propose a hierarchical controller for quadruped robot that could generate multiple gaits (i.e. pace, trot, bound) while tracking velocity command. Our controller is composed of two policies, each working as a central pattern generator and local feedback controller, and trained with hierarchical reinforcement learning. Experiment results show 1) the existence of optimal gait for specific velocity range 2) the efficiency of our hierarchical controller compared to a controller composed of a single policy, which usually shows a single gait. Codes are publicly available.

Motivation & Objective

• To address the limitation of single-policy reinforcement learning, which typically learns only one gait regardless of velocity command.
• To enable a quadruped robot to autonomously select and transition between multiple gaits (trot, pace, bound) based on desired velocity.
• To improve energy efficiency and velocity tracking performance by leveraging hierarchical control with CPG-based gait planning and local feedback.
• To demonstrate that optimal gait selection varies with velocity, as observed in biological quadrupeds.
• To provide a scalable, robust framework for multi-gait locomotion without relying on reference motion data or heuristic reward shaping.

Proposed method

• The controller uses a two-level hierarchical structure: a high-level CPG policy for gait selection and period modulation, and a low-level feedback policy for joint torque control.
• The high-level policy outputs CPG phase and period signals, which are used to generate rhythmic gait patterns across the four legs.
• The low-level policy uses CPG signals and real-time foot contact/environment feedback to compute joint torques via PD control and local feedback.
• A curriculum learning strategy is applied, gradually increasing the velocity command range during training to improve policy generalization.
• The reward function combines multiple cost terms: velocity tracking error (angular and linear), torque effort, joint speed, foot clearance, slip, orientation, smoothness, and leg phase consistency.
• The CPG signal is designed to allow phase shifts and period modulation, enabling transitions between gaits, though abrupt phase changes during transitions degrade tracking performance.

Experimental results

Research questions

• RQ1 Can a reinforcement learning controller learn multiple distinct gaits (trot, pace, bound) in response to varying velocity commands?
• RQ2 Does hierarchical control with a CPG-based high-level policy enable better energy efficiency and gait adaptability than a single-policy end-to-end approach?
• RQ3 How does the performance of the multi-gait controller compare to a single-gait baseline in terms of velocity tracking error and energy consumption?
• RQ4 What role does the CPG phase and period play in enabling smooth gait transitions and adaptive locomotion?
• RQ5 Can the controller learn optimal gait selection based on velocity, mirroring biological quadruped behavior?

Key findings

• The proposed hierarchical controller successfully learned and executed multiple gaits—trot, pace, and bound—across different velocity ranges, demonstrating adaptive gait selection.
• The controller achieved lower energy consumption in specific velocity ranges compared to the single-policy baseline, indicating improved efficiency.
• The single-policy baseline converged to a single gait (trot) regardless of command velocity, resulting in unnatural and suboptimal locomotion.
• Velocity tracking error was higher in the multi-gait controller due to abrupt CPG phase transitions during gait changes, indicating a need for smoother phase handling.
• The CPG-based controller showed robustness to contact disturbances and maintained stable locomotion across gaits, as confirmed by contact and CPG signal plots.
• The cost function design, including foot clearance, slip, and smoothness terms, contributed to stable and safe locomotion with minimal foot slippage and optimal clearance.