[Paper Review] Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior

Gabriel B. Margolis, Pulkit Agrawal|arXiv (Cornell University)|Dec 6, 2022

Robotic Locomotion and Control18 citations

TL;DR

This paper introduces Multiplicity of Behavior (MoB), a method to train a single reinforcement learning policy that generates diverse, controllable locomotion strategies—such as crouching, sprinting, or dancing—by tuning a few behavior parameters. The approach enables real-time adaptation to out-of-distribution environments like stairs or slippery terrain without retraining, demonstrating robust generalization across 10+ unseen tasks using a single open-source controller.

ABSTRACT

Learned locomotion policies can rapidly adapt to diverse environments similar to those experienced during training but lack a mechanism for fast tuning when they fail in an out-of-distribution test environment. This necessitates a slow and iterative cycle of reward and environment redesign to achieve good performance on a new task. As an alternative, we propose learning a single policy that encodes a structured family of locomotion strategies that solve training tasks in different ways, resulting in Multiplicity of Behavior (MoB). Different strategies generalize differently and can be chosen in real-time for new tasks or environments, bypassing the need for time-consuming retraining. We release a fast, robust open-source MoB locomotion controller, Walk These Ways, that can execute diverse gaits with variable footswing, posture, and speed, unlocking diverse downstream tasks: crouching, hopping, high-speed running, stair traversal, bracing against shoves, rhythmic dance, and more. Video and code release: https://gmargo11.github.io/walk-these-ways/

Motivation & Objective

• To address the limitation of standard RL policies that generalize poorly to out-of-distribution environments not seen during training.
• To overcome the need for slow, iterative retraining when policies fail in unseen scenarios such as stairs, uneven ground, or external shoves.
• To enable rapid, real-time tuning of locomotion behaviors via a single learned policy, reducing reliance on retraining.
• To demonstrate that a single policy can encode multiple effective strategies that generalize differently to unseen tasks.
• To provide a robust, open-source controller that supports diverse gaits and complex behaviors like dancing and high-speed leaping.

Proposed method

• Train a single quadruped locomotion policy in simulation using a fixed task (e.g., flat-ground walking), but with a behavior parameterization that controls gait features like footswing height, stance width, and frequency.
• Introduce a set of controllable behavior parameters (e.g., body height, footswing, gait frequency, phase offsets) that modulate the policy’s output in real time.
• Use a low-level controller that maps observations and behavior parameters to joint torques, enabling real-time behavior switching.
• Apply the same policy across diverse tasks by tuning parameters on the fly, including high-speed transitions and rhythmic gait sequences.
• Leverage human pilot tuning to explore behavior space in real time, bypassing the need for retraining.
• Release an open-source controller, Walk These Ways, that supports diverse gaits and complex behaviors such as agile leaps and synchronized dance routines.

Experimental results

Research questions

• RQ1 Can a single learned policy encode multiple effective locomotion strategies that generalize differently to unseen environments?
• RQ2 How effective is real-time behavior tuning via parameter modulation in enabling adaptation to out-of-distribution tasks like stairs or slippery surfaces?
• RQ3 To what extent does MoB improve generalization across diverse tasks without retraining?
• RQ4 Can behavior transitions be performed at high speed or with precise timing for complex tasks like parkour or synchronized dancing?
• RQ5 What is the trade-off between in-distribution performance and the flexibility provided by MoB?

Key findings

• A single policy trained only on flat ground successfully generalizes to diverse out-of-distribution tasks, including stairs, uneven terrain, and external shoves, via real-time behavior tuning.
• The controller enables high-speed gait transitions, such as accelerating to 3 m/s and performing a 60 cm forward leap, demonstrating agility in dynamic maneuvers.
• Precise timing of gait parameters allows for synchronized dance routines at 90 bpm, with footstep gaps aligned to musical beats using phase and frequency modulation.
• The system supports diverse behaviors including crouching, hopping, high-speed running, bracing against shoves, and payload manipulation.
• Despite a slight trade-off in flat-ground sprinting performance, the method enables robust adaptation across 10+ unseen tasks using only parameter tuning.
• The open-source Walk These Ways controller supports real-time behavior selection and has been validated in both simulation and real-world deployment.