[Paper Review] RL + Model-Based Control: Using On-Demand Optimal Control to Learn Versatile Legged Locomotion

Dong‐Ho Kang, Jin Cheng+3 more|arXiv (Cornell University)|May 29, 2023

Robotic Locomotion and Control1 citations

TL;DR

This paper proposes a hybrid control framework that combines model-based optimal control (MBOC) with reinforcement learning (RL) to enable versatile, robust legged locomotion. By generating on-demand reference motions via finite-horizon optimal control and training an RL policy to imitate them, the method achieves high generalization to complex terrains and diverse gaits without robot-specific reward engineering, outperforming model-predictive control in rough environments.

ABSTRACT

This paper presents a control framework that combines model-based optimal control and reinforcement learning (RL) to achieve versatile and robust legged locomotion. Our approach enhances the RL training process by incorporating on-demand reference motions generated through finite-horizon optimal control, covering a broad range of velocities and gaits. These reference motions serve as targets for the RL policy to imitate, leading to the development of robust control policies that can be learned with reliability. Furthermore, by utilizing realistic simulation data that captures whole-body dynamics, RL effectively overcomes the inherent limitations in reference motions imposed by modeling simplifications. We validate the robustness and controllability of the RL training process within our framework through a series of experiments. In these experiments, our method showcases its capability to generalize reference motions and effectively handle more complex locomotion tasks that may pose challenges for the simplified model, thanks to RL's flexibility. Additionally, our framework effortlessly supports the training of control policies for robots with diverse dimensions, eliminating the necessity for robot-specific adjustments in the reward function and hyperparameters.

Motivation & Objective

• To overcome the limitations of purely model-based optimal control, which relies on simplifying assumptions that restrict versatility and adaptability.
• To address the challenges of standard RL in legged locomotion, particularly the need for extensive reward shaping and lack of behavioral priors.
• To create a unified framework that leverages the transparency and efficiency of MBOC for motion generation and the flexibility and generalization capacity of RL for policy learning.
• To enable robust sim-to-real transfer and adaptability to diverse robot morphologies without requiring robot-specific reward functions or hyperparameter tuning.

Proposed method

• Generate on-demand reference motions using finite-horizon optimal control (OCP) with a simplified dynamics model (e.g., VHIPM) to cover a wide range of velocities and gaits.
• Train an RL policy to imitate these MBOC-generated reference motions using a carefully designed imitation reward function that balances fidelity and generalization.
• Integrate realistic simulation dynamics (including full-body dynamics) to allow the RL policy to overcome modeling inaccuracies inherent in the MBOC framework.
• Apply extensive randomization during training—on terrain friction, Perlin-noise-based uneven terrain, gravity vector perturbations, and external impulses—to enhance robustness and sim-to-real transfer.
• Use a contact-phase parameterization that supports non-periodic and asymmetric gaits, enabling diverse locomotion behaviors beyond symmetric patterns.
• Leverage the same MBOC demonstrations across different quadruped robots (Go1 and Aliengo), enabling transferability without re-tuning reward functions or hyperparameters.

Experimental results

Research questions

• RQ1 Can on-demand optimal control generate diverse, dynamically coherent reference motions that serve as effective targets for RL imitation in legged locomotion?
• RQ2 To what extent can an RL policy generalize beyond the simplifications of the MBOC model to handle complex, real-world scenarios like uneven terrain and external perturbations?
• RQ3 How effectively can the hybrid framework achieve robust sim-to-real transfer without robot-specific reward engineering or hyperparameter tuning?
• RQ4 Can the framework support versatile locomotion behaviors—including non-periodic and asymmetric gaits—while maintaining stability and controllability?
• RQ5 What is the trade-off between strict imitation of MBOC motions and the flexibility needed for RL generalization in complex environments?

Key findings

• The RL policy successfully generalizes MBOC reference motions to navigate rough, uneven terrain and inclined surfaces, outperforming a model-predictive control (MPC) baseline in a 5 m traversal task within 10 seconds.
• The policy maintains distinct gait styles and responds accurately to arbitrary velocity commands without requiring task-specific reward shaping or engineering interventions.
• Hardware experiments on both the Unitree Go1 and Aliengo quadruped robots confirmed robust performance under external perturbations, uneven terrain, and variable friction conditions.
• The framework enables effective sim-to-real transfer through systematic randomization of terrain, friction, gravity, and external impulses during training, with no need for robot-specific reward adjustments.
• The method achieves reliable training and high policy performance (16.4M samples, 46 hours on a 64-core CPU) while maintaining versatility across diverse locomotion tasks.
• The approach generalizes beyond the constraints of periodic or symmetric gaits, demonstrating potential for animal-like locomotion behaviors through contact-phase parameterization.