[Paper Review] Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots

Cheng Chi, Zhenjia Xu+6 more|arXiv (Cornell University)|Feb 15, 2024

Modular Robots and Swarm Intelligence4 citations

TL;DR

UMI introduces a portable, low-cost framework that enables direct transfer of in-the-wild human demonstrations to deployable robot policies using hand-held grippers with a fisheye camera and IMU. By employing latency-matched inference, relative-trajectory action representation, and Diffusion Policy, UMI achieves zero-shot generalization across diverse, dynamic, bimanual, and long-horizon tasks on multiple robot platforms.

ABSTRACT

We present Universal Manipulation Interface (UMI) -- a data collection and policy learning framework that allows direct skill transfer from in-the-wild human demonstrations to deployable robot policies. UMI employs hand-held grippers coupled with careful interface design to enable portable, low-cost, and information-rich data collection for challenging bimanual and dynamic manipulation demonstrations. To facilitate deployable policy learning, UMI incorporates a carefully designed policy interface with inference-time latency matching and a relative-trajectory action representation. The resulting learned policies are hardware-agnostic and deployable across multiple robot platforms. Equipped with these features, UMI framework unlocks new robot manipulation capabilities, allowing zero-shot generalizable dynamic, bimanual, precise, and long-horizon behaviors, by only changing the training data for each task. We demonstrate UMI's versatility and efficacy with comprehensive real-world experiments, where policies learned via UMI zero-shot generalize to novel environments and objects when trained on diverse human demonstrations. UMI's hardware and software system is open-sourced at https://umi-gripper.github.io.

Motivation & Objective

• Address the challenge of transferring complex, real-world human manipulation skills to robots without requiring in-the-wild robot hardware.
• Overcome limitations in prior hand-held gripper systems, such as insufficient visual context, action imprecision, latency mismatches, and inadequate policy representation.
• Enable scalable, decentralized data collection using non-expert demonstrators via a low-cost, intuitive interface.
• Develop a hardware-agnostic policy learning framework that generalizes zero-shot to novel environments and objects.

Proposed method

• Use a hand-held gripper equipped with a GoPro camera (155° fisheye lens) and side mirrors to enhance visual context and enable implicit stereo tracking.
• Leverage the GoPro’s IMU and a custom SLAM system to recover precise 6-DOF trajectories from monocular video, reducing scale ambiguity and motion blur effects.
• Implement inference-time latency matching to align recorded demonstration latencies with real-time system latencies during policy inference.
• Represent actions as relative trajectories instead of absolute poses to decouple policy learning from global coordinate precision.
• Apply Diffusion Policy to model multimodal action distributions inherent in diverse human demonstrations.
• Filter collected data for kinematic feasibility to ensure downstream policy deployment on real robots.

Experimental results

Research questions

• RQ1 Can a low-cost, portable hand-held gripper system with minimal sensors enable high-fidelity action recovery for complex manipulation tasks?
• RQ2 How can latency mismatches between demonstration and inference be mitigated to ensure policy robustness in real-time deployment?
• RQ3 To what extent can relative-trajectory action representation improve policy generalization compared to absolute pose regression?
• RQ4 Can a single policy architecture, trained on diverse in-the-wild human demonstrations, achieve zero-shot generalization across novel objects and environments?
• RQ5 How does the visual context provided by a wide-angle fisheye lens and side mirrors improve action recovery accuracy and policy performance?

Key findings

• UMI’s SLAM system achieves a mean Absolute Trajectory Error (ATE) of 6.1 mm for position and 3.5° for rotation, with a relative pose error of 10.1 mm and 0.8° between two grippers.
• On the cup arrangement task, UMI data collection is more than 3× faster than teleoperation and reaches 48% of human hand speed.
• On the dynamic tossing task, UMI achieves 64% of human hand speed, while teleoperation failed to produce a single successful demonstration in 15 minutes.
• Policies trained with UMI zero-shot generalize to novel environments and objects, including unseen objects and layouts, without retraining.
• The framework enables deployment of complex behaviors such as bimanual sweater folding, dynamic object tossing, and precise dish washing using only human demonstration data.
• UMI’s data collection is scalable and accessible, with all data stored in a single MP4 file, enabling geographically distributed, non-expert participation.