[Paper Review] Open X-Embodiment: Robotic Learning Datasets and RT-X Models

Open X-Embodiment Collaboration, O'Neill, Abby+292 more|arXiv (Cornell University)|Oct 13, 2023

Reinforcement Learning in RoboticsComputer Science101 citations

TL;DR

This work introduces Open X-Embodiment, a 1M+ trajectory robotic dataset across 22 embodiments and the RT-X models that transfer knowledge across robots, enabling positive transfer and improved generalization.

ABSTRACT

Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train generalist X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. More details can be found on the project website https://robotics-transformer-x.github.io.

Motivation & Objective

• Motivate the need for X-embodiment data to enable generalist robotic policies akin to NLP/vision models.
• Provide a standardized, large-scale multi-robot dataset spanning many embodiments and tasks.
• Evaluate RT-1-X and RT-2-X policies trained on multi-robot data for transfer and generalization.
• Offer open-source data formats, baselines, and pretrained RT-X checkpoints to energize the community.

Proposed method

• Assemble a dataset from 22 robot embodiments collected across 21 institutions into a unified Open X-Embodiment Dataset consisting of 1M+ trajectories.
• Adopt a coarse alignment of observation and action spaces to a common 7-DoF end-effector action representation.
• Evaluate two Transformer-based policy architectures (RT-1-X and RT-2-X) on multi-embodiment data.
• Train RT-1-X on robotics data only, and RT-2-X via co-fine-tuning with both robotics data and web-scale vision-language data.
• Use cross-entropy objectives over discrete action tokens for RT-1-X and RT-2-X.
• Assess performance in in-distribution and out-of-distribution settings, with ablations on history length and web pretraining.

Figure 0 : The Open X-Embodiment Dataset. (a) : the dataset consists of 60 individual datasets across $22$ embodiments. (b) : the Franka robot has the largest diversity in visually distinct scenes due to the large number of Franka datasets, (c) : xArm and Google Robot contribute the most number of t

Experimental results

Research questions

• RQ1 Does training on multi-embodiment data yield positive transfer to individual robots?
• RQ2 Does multi-robot exposure improve generalization to unseen tasks, objects, and environments?
• RQ3 How do model size, history, and web pretraining influence XY transfer and emergent skills across embodiments?

Key findings

• RT-1-X achieves up to 50% higher mean success rate than the Original Method or RT-1 on targeted in-distribution tasks.
• RT-2-X (55B) achieves approximately 3× generalization improvements over models trained only on the evaluation embodiment.
• Co-training on multi-robot data enables emergent skills that transfer to other robots (e.g., Google Robot improves with Bridge data from WidowX).
• Larger model capacity (55B RT-2-X) and web-based pretraining are critical for strong performance and generalization in data-rich domains.
• Shorter history hurts generalization, while including a short image history and web pretraining significantly boosts results.

Figure 1 : RT-1-X and RT-2-X both take images and a text instruction as input and output discretized end-effector actions. RT-1-X is an architecture designed for robotics, with a FiLM [ 116 ] conditioned EfficientNet [ 117 ] and a Transformer [ 118 ] . RT-2-X builds on a VLM backbone by representing