[Paper Review] Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning

Tianyu Li, Nathan Lambert+3 more|arXiv (Cornell University)|Sep 26, 2019

Reinforcement Learning in Robotics37 references4 citations

TL;DR

This paper proposes a hierarchical reinforcement learning framework that combines model-free learning of cyclic locomotion primitives with model-based planning using coarse dynamics models to enable sample-efficient, generalizable goal-directed locomotion on real-world robots. The approach allows a 18-DOF hexapod robot to learn from scratch to reach goals up to 12 meters away in just two hours of hardware training, demonstrating robustness to noise and environmental variation.

ABSTRACT

Learning to locomote to arbitrary goals on hardware remains a challenging problem for reinforcement learning. In this paper, we present a hierarchical learning framework that improves sample-efficiency and generalizability of locomotion skills on real-world robots. Our approach divides the problem of goal-oriented locomotion into two sub-problems: learning diverse primitives skills, and using model-based planning to sequence these skills. We parametrize our primitives as cyclic movements, improving sample-efficiency of learning on a 18 degrees of freedom robot. Then, we learn coarse dynamics models over primitive cycles and use them in a model predictive control framework. This allows us to learn to walk to arbitrary goals up to 12m away, after about two hours of training from scratch on hardware. Our results on a Daisy hexapod hardware and simulation demonstrate the efficacy of our approach at reaching distant targets, in different environments and with sensory noise.

Motivation & Objective

• To address the challenge of sample inefficiency and poor generalization in reinforcement learning for real-world legged robot locomotion.
• To enable learning of goal-directed locomotion policies that generalize to arbitrary, unseen targets on hardware.
• To reduce reliance on simulation-to-real transfer by training directly on hardware with improved data efficiency.
• To develop a scalable framework that separates low-level skill learning from high-level goal planning.
• To improve robustness to sensory noise and environmental changes during real-world deployment.

Proposed method

• Parametrize low-level locomotion policies as cyclic movements (e.g., walking forward, turning) to improve sample efficiency during training.
• Use model-free reinforcement learning (Soft Actor-Critic) to train these cyclic policies directly on hardware.
• Learn coarse dynamics models over one cycle of primitive actions using only 50 hardware samples per primitive.
• Employ model predictive control (MPC) with the coarse dynamics models to plan sequences of primitives for reaching arbitrary goals.
• Introduce an orientation-guided reward in the high-level controller to correct for drift in global position sensing.
• Decompose the control problem into two stages: learning diverse primitives first, then planning their sequence using learned dynamics models.

Figure 1 : The hexapod Daisy used in the experiments. Using our hierarchical control framework, Daisy learned from scratch to reach goals as far as 12 meters in 2 hours of training.

Experimental results

Research questions

• RQ1 Can a hierarchical RL framework improve sample efficiency and generalization in real-world locomotion learning?
• RQ2 Can cyclic parametrization of locomotion policies enhance learning efficiency on high-DOF hardware robots?
• RQ3 Can coarse dynamics models trained on minimal hardware data enable effective long-horizon planning for arbitrary goals?
• RQ4 How well does the framework generalize to distant targets, different terrains, and sensory noise in real-world settings?
• RQ5 Can the combination of model-free and model-based components outperform purely model-free or model-based approaches in real-world locomotion?

Key findings

• The Daisy hexapod robot successfully learned to reach goals up to 12 meters away from its starting position after only two hours of direct hardware training.
• The system generalized robustly to different environments, including varied flooring and sensory noise, without retraining.
• The high-level controller with orientation guidance corrected for position drift, enabling accurate goal reaching despite tracking errors.
• The framework achieved reliable performance on both single-goal and sequential waypoint tasks in hardware experiments.
• The use of coarse dynamics models trained on just 50 samples per primitive enabled efficient and effective planning in primitive space.
• The approach demonstrated the first successful hybrid model-free and model-based RL framework for goal-directed locomotion on real-world legged robots.

Figure 2 : Hierarchical Control Flow Chart