[Paper Review] Learning to Walk in the Real World with Minimal Human Effort

Sehoon Ha, Peng Xu+3 more|arXiv (Cornell University)|Feb 20, 2020

Robotic Locomotion and Control57 references76 citations

TL;DR

The paper presents an autonomous real-world RL system for legged locomotion that uses multi-task learning and safety-constrained SAC to train walking policies on multiple terrains with minimal human intervention, achieving efficient real-world learning on Minitaur and reducing falls.

ABSTRACT

Reliable and stable locomotion has been one of the most fundamental challenges for legged robots. Deep reinforcement learning (deep RL) has emerged as a promising method for developing such control policies autonomously. In this paper, we develop a system for learning legged locomotion policies with deep RL in the real world with minimal human effort. The key difficulties for on-robot learning systems are automatic data collection and safety. We overcome these two challenges by developing a multi-task learning procedure and a safety-constrained RL framework. We tested our system on the task of learning to walk on three different terrains: flat ground, a soft mattress, and a doormat with crevices. Our system can automatically and efficiently learn locomotion skills on a Minitaur robot with little human intervention. The supplemental video can be found at: \url{https://youtu.be/cwyiq6dCgOc}.

Motivation & Objective

• Reduce human effort required for real-world legged robot learning by automating data collection, resets, and safety in training.
• Enable learning of multiple locomotion skills (forward, backward, turns) within a single training session.
• Demonstrate robust learning across diverse terrains (flat ground, soft mattress, crevassed doormat).
• Show that safety-constrained RL reduces falls and training interruptions in real-world data collection.

Proposed method

• Adopt a multi-task RL framework with a task scheduler that selects walking directions to keep the robot within the training workspace.
• Define per-task rewards that encode desired direction of motion relative to episode start with a three-dimensional task vector w^i.
• Train separate SAC policies for each task without shared actors/critics to avoid cross-task interference.
• Introduce a safety constraint as a constrained MDP using a Lagrangian multiplier to bound torso pitch/roll during training.
• Optimize using dual gradient descent to jointly update policy and the Lagrangian multiplier.
• Use early termination near workspace boundaries with safety-aware return calculations to prevent excessive resets.

Experimental results

Research questions

• RQ1 Can autonomous real-world RL learn legged locomotion with minimal human intervention on multiple terrains?
• RQ2 Is it feasible to train multiple locomotion policies simultaneously in real-world settings?
• RQ3 Does a safety-constrained RL framework reduce the number of falls and training disruptions compared to standard methods?

Key findings

• The system learns to walk on flat ground, a soft mattress, and a doormat with crevices with minimal human intervention (zero manual resets in two of three flat-ground runs).
• Four locomotion policies (forward, backward, turn left, turn right) can be learned in a single session, enabling a complete directional walking controller at test time.
• On flat terrain, training two policies (forward and backward) required about 1.5 hours (~60k steps per policy).
• On challenging surfaces, forward/backward walking learned in 5.5 hours ( mattress, ~200k steps) and 4.5 hours (doormat, ~150k steps).
• The safety-constrained SAC reduces falls substantially compared with SAC without safety constraints (roughly 40 falls vs. >100 in a baseline with no safety constraint), reducing training interruptions.
• Multi-task learning significantly lowers out-of-workspace failures to about 5-10% of the baseline across workspace sizes.