[Paper Review] Robust and Versatile Bipedal Jumping Control through Reinforcement Learning

Zhongyu Li, Xue Bin Peng+4 more|arXiv (Cornell University)|Feb 19, 2023

Robotic Locomotion and Control4 citations

TL;DR

This paper presents a reinforcement learning framework that enables a torque-controlled bipedal robot, Cassie, to perform robust and versatile dynamic jumps in the real world. By using a goal-conditioned policy with a novel long- and short-term I/O history encoder and a multi-stage training scheme, the robot learns to jump to diverse targets—such as different distances, directions, and elevated platforms—achieving high success rates and emergent recovery behaviors from perturbations.

ABSTRACT

This work aims to push the limits of agility for bipedal robots by enabling a torque-controlled bipedal robot to perform robust and versatile dynamic jumps in the real world. We present a reinforcement learning framework for training a robot to accomplish a large variety of jumping tasks, such as jumping to different locations and directions. To improve performance on these challenging tasks, we develop a new policy structure that encodes the robot's long-term input/output (I/O) history while also providing direct access to a short-term I/O history. In order to train a versatile jumping policy, we utilize a multi-stage training scheme that includes different training stages for different objectives. After multi-stage training, the policy can be directly transferred to a real bipedal Cassie robot. Training on different tasks and exploring more diverse scenarios lead to highly robust policies that can exploit the diverse set of learned maneuvers to recover from perturbations or poor landings during real-world deployment. Such robustness in the proposed policy enables Cassie to succeed in completing a variety of challenging jump tasks in the real world, such as standing long jumps, jumping onto elevated platforms, and multi-axes jumps.

Motivation & Objective

• Enable a life-sized, torque-controlled bipedal robot to perform agile, dynamic jumps in the real world, overcoming the limitations of prior model-based and single-task RL methods.
• Address the challenge of controlling complex, hybrid dynamics in bipedal jumping, including takeoff, flight, and high-impact landings within seconds.
• Develop a goal-conditioned policy that generalizes across diverse jumping tasks (e.g., forward, lateral, turning, elevated jumps) to enhance robustness and adaptability.
• Improve sim-to-real transfer and robustness by training on a wide variety of jumping tasks rather than relying solely on dynamics randomization.
• Demonstrate that versatility in training leads to emergent recovery behaviors during real-world deployment, such as switching to a hop after a poor landing.

Proposed method

• Propose a novel policy architecture with a dual-encoder structure: a long-term I/O history encoder to capture past trajectories and a short-term encoder for immediate feedback, both jointly trained with the policy.
• Implement a multi-stage training scheme that decomposes the complex multi-goal jumping problem into three sub-problems: (1) learning basic jumps, (2) learning diverse jump targets, and (3) fine-tuning for robustness and generalization.
• Use goal-conditioned reinforcement learning to train a single policy that maps state and goal (e.g., target position, orientation) to torque commands, enabling zero-shot deployment to new targets.
• Integrate a temporal encoder that learns to extract relevant information from the robot’s own I/O history—such as impact forces and contact transitions—without relying on pre-specified environment parameters.
• Leverage simulation-to-real transfer by training in a high-fidelity MuJoCo simulation with randomization of dynamics and goal distributions, then deploying directly to the real Cassie robot.
• Apply curriculum learning and curriculum-based goal scheduling to progressively increase task diversity and difficulty during training, improving sample efficiency and policy generalization.

Experimental results

Research questions

• RQ1 Can a single, goal-conditioned reinforcement learning policy enable a torque-controlled bipedal robot to perform a wide variety of dynamic jumps in the real world, including jumps to different locations, directions, and elevations?
• RQ2 How does multi-goal training—learning multiple jumping tasks simultaneously—affect the robustness of the policy during real-world deployment, especially under perturbations or poor landings?
• RQ3 To what extent does incorporating long-term and short-term input/output history into the policy architecture improve learning efficiency and performance on complex, high-impact jumping tasks?
• RQ4 Can a policy trained in simulation with diverse goals generalize directly to the real world without fine-tuning, and what role does task diversity play in this sim-to-real transfer?
• RQ5 Does training on multiple jumping tasks lead to emergent recovery behaviors not explicitly programmed, such as switching to a hop after a failed landing?

Key findings

• The proposed method achieved a maximum forward leap distance of 1.4 meters on the real Cassie robot, significantly outperforming prior work that only reached ~0.5 meters.
• The robot successfully performed standing long jumps (up to 1.4m forward), lateral jumps (±0.3m), turning jumps (±55°), and jumps onto elevated platforms (0.44m height), demonstrating high versatility.
• The policy achieved a 90% success rate on diverse jumping tasks in simulation and maintained over 80% success rate in real-world deployment, indicating strong sim-to-real transfer capability.
• Robustness emerged from multi-goal training: the policy could recover from large landing impacts and external perturbations by switching to a different learned maneuver, such as a hop, without re-planning.
• Ablation studies showed that multi-goal training significantly improved robustness compared to single-task training with or without dynamics randomization, even under added perturbations.
• The dual-encoder architecture (long- and short-term I/O history) outperformed prior methods like RMA/TS, which require separate teacher-student training and finetuning, by enabling more flexible and direct use of historical I/O data.