[Paper Review] One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion

Nico Bohlinger, Grzegorz Czechmanowski+5 more|arXiv (Cornell University)|Sep 10, 2024

Sociology and Norbert Elias4 citations

TL;DR

This paper proposes URMA, a unified end-to-end deep reinforcement learning framework that enables a single policy to control diverse legged robot morphologies—such as quadrupeds, hexapods, bipeds, and humanoids—through morphology-agnostic encoders and decoders. The method achieves zero-shot transfer to unseen robots in both simulation and the real world, demonstrating robust generalization across morphological variations without retraining.

ABSTRACT

Deep Reinforcement Learning techniques are achieving state-of-the-art results in robust legged locomotion. While there exists a wide variety of legged platforms such as quadruped, humanoids, and hexapods, the field is still missing a single learning framework that can control all these different embodiments easily and effectively and possibly transfer, zero or few-shot, to unseen robot embodiments. We introduce URMA, the Unified Robot Morphology Architecture, to close this gap. Our framework brings the end-to-end Multi-Task Reinforcement Learning approach to the realm of legged robots, enabling the learned policy to control any type of robot morphology. The key idea of our method is to allow the network to learn an abstract locomotion controller that can be seamlessly shared between embodiments thanks to our morphology-agnostic encoders and decoders. This flexible architecture can be seen as a potential first step in building a foundation model for legged robot locomotion. Our experiments show that URMA can learn a locomotion policy on multiple embodiments that can be easily transferred to unseen robot platforms in simulation and the real world.

Motivation & Objective

• To address the lack of a unified learning framework that can control multiple legged robot morphologies without retraining.
• To enable zero-shot deployment of a single locomotion policy to novel robot embodiments in simulation and the real world.
• To develop a morphology-agnostic neural architecture that handles variable-sized observation and action spaces across different robot platforms.
• To advance toward a foundation model for legged robot locomotion by enabling knowledge transfer across diverse morphologies.
• To overcome the limitations of existing methods that require task-specific architectures and hyperparameters for each robot type.

Proposed method

• Introduces URMA, a Unified Robot Morphology Architecture with morphology-agnostic encoders and decoders to standardize input and output representations across different robot morphologies.
• Employs an end-to-end multi-task reinforcement learning setup where a single policy is trained on 16 diverse robot platforms simultaneously in simulation.
• Uses domain randomization and dynamic environment randomization during training to improve robustness and enable zero-shot transfer.
• Applies a shared representation space that abstracts from specific joint counts and kinematic structures, allowing the policy to generalize across morphologies.
• Leverages a hierarchical architecture with shared feature extractors and task-specific heads that are dynamically adapted via attention or routing mechanisms.
• Employs curriculum learning and reward shaping to stabilize training across diverse and complex morphologies.

Figure 1: Top – We train a single locomotion policy for multiple robot embodiments in simulation. Bottom – We can transfer and deploy the policy on three real-world platforms by randomizing the embodiments and environment dynamics during training.

Experimental results

Research questions

• RQ1 Can a single deep reinforcement learning policy be trained to control multiple distinct legged robot morphologies with varying numbers of limbs and joints?
• RQ2 To what extent can a policy trained on diverse morphologies generalize to entirely unseen robot platforms without fine-tuning?
• RQ3 How effective is the proposed morphology-agnostic architecture in enabling zero-shot deployment across simulation and real-world robots?
• RQ4 What is the impact of shared representation learning on policy generalization and sample efficiency across morphological variations?
• RQ5 Can the framework serve as a foundation model for legged robot locomotion, enabling few-shot adaptation to new platforms?

Key findings

• The URMA framework successfully trained a single policy on 16 different robot morphologies, including quadrupeds, hexapods, bipeds, and humanoids, in simulation.
• The trained policy achieved zero-shot deployment on two unseen simulated robot platforms and three real-world robots without any retraining or hyperparameter tuning.
• The policy demonstrated robust locomotion performance across diverse terrains and dynamic conditions in both simulation and real-world deployment.
• The morphology-agnostic architecture enabled seamless handling of variable-sized observation and action spaces, eliminating the need for architecture reconfiguration per robot type.
• The method achieved stable and efficient learning across diverse morphologies, with performance comparable to task-specific policies trained from scratch.
• The framework shows strong potential as a foundation model for legged robot locomotion, enabling rapid deployment to new platforms with minimal adaptation.

Figure 2: Overview of URMA. Left – Joint observations and descriptions are encoded and combined into a single joint latent vector through an attention head. Bottom center – Feet observations and descriptions are encoded in the same way. Top center – Joint latent, feet latent, and general observation