[Paper Review] Learning Semantics-Aware Locomotion Skills from Human Demonstration

Yuxiang Yang, Xiangyun Meng+4 more|arXiv (Cornell University)|Jun 27, 2022

Robotic Locomotion and Control4 citations

TL;DR

This paper presents a hierarchical framework that learns semantics-aware locomotion skills for quadrupedal robots using only 40 minutes of human demonstration data. By combining pre-trained semantic perception, imitation learning for speed selection, and a gait selector, the system enables safe, efficient traversal of complex off-road terrains—achieving over 6km of failure-free navigation across diverse, unseen trails.

ABSTRACT

The semantics of the environment, such as the terrain type and property, reveals important information for legged robots to adjust their behaviors. In this work, we present a framework that learns semantics-aware locomotion skills from perception for quadrupedal robots, such that the robot can traverse through complex offroad terrains with appropriate speeds and gaits using perception information. Due to the lack of high-fidelity outdoor simulation, our framework needs to be trained directly in the real world, which brings unique challenges in data efficiency and safety. To ensure sample efficiency, we pre-train the perception model with an off-road driving dataset. To avoid the risks of real-world policy exploration, we leverage human demonstration to train a speed policy that selects a desired forward speed from camera image. For maximum traversability, we pair the speed policy with a gait selector, which selects a robust locomotion gait for each forward speed. Using only 40 minutes of human demonstration data, our framework learns to adjust the speed and gait of the robot based on perceived terrain semantics, and enables the robot to walk over 6km without failure at close-to-optimal speed.

Motivation & Objective

• Enable legged robots to proactively adapt locomotion based on terrain semantics (e.g., grass, mud, asphalt) rather than relying solely on geometric features.
• Address the challenge of sample efficiency and safety in real-world policy learning due to high data cost and hardware risk in off-road environments.
• Develop a data-efficient, safe training framework that avoids direct reinforcement learning exploration in the real world.
• Improve traversability and robustness by jointly learning speed and gait selection based on perceived terrain semantics.
• Achieve generalization across unseen off-road terrains with minimal human demonstration data.

Proposed method

• Pre-train a semantic segmentation network on an off-road driving dataset to extract semantic embeddings for terrain understanding.
• Fine-tune the perception model using a hidden-layer representation from the pre-trained network to improve generalization and sample efficiency.
• Use human demonstrations to collect speed choices from RGB images, then train a speed policy via imitation learning to select appropriate forward speeds.
• Implement a hierarchical control architecture where the speed policy is paired with a gait selector that chooses a robust gait for each selected speed.
• Deploy the end-to-end framework on a Unitree A1 robot for real-world deployment without simulation.
• Leverage the semantic embedding and imitation learning to enable real-world policy training with minimal exploration and high safety.

Experimental results

Research questions

• RQ1 Can a legged robot learn to adjust its locomotion speed and gait based on semantic terrain categories using only human demonstrations?
• RQ2 How can semantic perception improve locomotion performance in unstructured off-road environments compared to geometric-only approaches?
• RQ3 To what extent can imitation learning with pre-trained perception reduce data requirements and ensure safety in real-world robot learning?
• RQ4 Can the framework generalize to unseen terrains and achieve high-speed, failure-free traversal across diverse outdoor trails?
• RQ5 What is the impact of different perception fine-tuning strategies (e.g., using hidden-layer features vs. class predictions) on policy performance?

Key findings

• The framework successfully enables a quadruped robot to traverse over 6km of diverse, unseen outdoor trails without any failures, demonstrating strong generalization.
• Using only 40 minutes of human demonstration data, the system learns to select appropriate speeds and gaits based on terrain semantics, achieving high-speed locomotion on flat, rigid surfaces like asphalt.
• The robot outperforms the manufacturer’s default controller in both speed and safety, particularly on challenging terrains such as rocks and pebbles.
• The gait selector significantly improves stability, reducing slips and failures compared to a gait-only policy, which slipped twice on rocky terrain.
• The ablation study confirms that fine-tuning from a pre-trained hidden-layer embedding yields the most accurate speed predictions and lowest validation error.
• The policy selects faster speeds on rigid, flat terrains (e.g., asphalt) and slower, more cautious gaits on deformable or uneven terrains (e.g., grass, mud), aligning with human intuition and terrain properties.