[Paper Review] A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning

Laura Smith, Ilya Kostrikov+1 more|arXiv (Cornell University)|Aug 16, 2022

Robotic Locomotion and ControlEngineering25 citations

TL;DR

The paper demonstrates real-world learning of quadruped locomotion using model-free deep RL, achieving walking in about 20 minutes across various terrains without novel algorithms. It emphasizes careful task design and implementation over new algorithmic advances.

ABSTRACT

Deep reinforcement learning is a promising approach to learning policies in uncontrolled environments that do not require domain knowledge. Unfortunately, due to sample inefficiency, deep RL applications have primarily focused on simulated environments. In this work, we demonstrate that the recent advancements in machine learning algorithms and libraries combined with a carefully tuned robot controller lead to learning quadruped locomotion in only 20 minutes in the real world. We evaluate our approach on several indoor and outdoor terrains which are known to be challenging for classical model-based controllers. We observe the robot to be able to learn walking gait consistently on all of these terrains. Finally, we evaluate our design decisions in a simulated environment.

Motivation & Objective

• Demonstrate that fully model-free deep RL can learn quadrupedal walking directly in the real world in under 20 minutes.
• Show robustness of walking across diverse indoor and outdoor terrains without novel algorithmic components.
• Identify design choices and system factors that enable sample-efficient real-world RL for locomotion.

Proposed method

• Use off-policy actor-critic RL based on SAC/DroQ style frameworks with regularization (dropout, layer normalization) to enable high update-to-data ratios.
• Employ low-level action space of PD position targets for 12 joints at 20 Hz.
• Define state as root orientation/velocity, joint states, foot contacts, and previous action; actions are PD targets.
• Provide a simple velocity-based reward within a target velocity interval, projected onto the ground plane.
• Train synchronously on a real robot (A1) using JAX with an ensemble of Q-functions and target networks.
• Compare various regularization/normalization strategies to achieve sample efficiency rather than seeking a new algorithmic component.

Experimental results

Research questions

• RQ1 Can fully model-free deep RL learn quadrupedal locomotion directly in the real world within a short wall-clock time?
• RQ2 How do design choices (action space, rewards, regularization) influence sample efficiency and training stability in real-world RL for locomotion?
• RQ3 Is real-world policy learning of locomotion robust across multiple terrains and environments without simulation-to-real transfer?
• RQ4 What is the impact of synchronous per-step updates versus episodic updates on real-world training efficiency?

Key findings

• Real-world walking can be learned in under 20 minutes (approximately 20 total minutes of wall-clock time) using standard model-free RL with careful design.
• Training across five terrains including indoor/outdoor settings yields effective gaits without novel algorithmic components.
• Various regularization/normalization strategies (e.g., layer normalization, dropout) enable higher update-to-data ratios, improving sample efficiency.
• Synchronous, per-step updates are feasible and beneficial for real-time training on a laptop with a GPU.
• Constraining the action space and choosing PD targets for joints are crucial for stable learning in the real world.