[Paper Review] XSkill: Cross Embodiment Skill Discovery

Mengda Xu, Zhenjia Xu+3 more|arXiv (Cornell University)|Jul 19, 2023

Human Pose and Action RecognitionComputer Science3 citations

TL;DR

XSkill proposes a cross-embodiment skill discovery framework that learns shared skill prototypes from unlabeled human and robot manipulation videos using self-supervised representation learning. It transfers these skills to a robot via a conditional diffusion policy and composes them for unseen tasks using a single human demonstration, achieving robust zero-shot generalization in both simulation and real-world environments with state-of-the-art performance.

ABSTRACT

Human demonstration videos are a widely available data source for robot learning and an intuitive user interface for expressing desired behavior. However, directly extracting reusable robot manipulation skills from unstructured human videos is challenging due to the big embodiment difference and unobserved action parameters. To bridge this embodiment gap, this paper introduces XSkill, an imitation learning framework that 1) discovers a cross-embodiment representation called skill prototypes purely from unlabeled human and robot manipulation videos, 2) transfers the skill representation to robot actions using conditional diffusion policy, and finally, 3) composes the learned skill to accomplish unseen tasks specified by a human prompt video. Our experiments in simulation and real-world environments show that the discovered skill prototypes facilitate both skill transfer and composition for unseen tasks, resulting in a more general and scalable imitation learning framework. The benchmark, code, and qualitative results are on https://xskill.cs.columbia.edu/

Motivation & Objective

• To address the challenge of transferring reusable manipulation skills from unstructured human demonstration videos to robots with different physical embodiments.
• To discover a shared, cross-embodiment skill representation space without requiring labeled demonstrations or task-specific supervision.
• To enable zero-shot generalization to unseen tasks by composing discovered skills using only a single human demonstration video as input.
• To reduce reliance on costly, expert-annotated robot demonstration data by leveraging widely available human videos.
• To develop a scalable and general imitation learning framework that supports skill transfer and composition across diverse embodiments.

Proposed method

• XSkill learns a shared embedding space for human and robot skills using self-supervised contrastive learning on unaligned video clips, where similar action effects (i.e., skills) are pulled closer in feature space.
• It introduces learnable skill prototypes as cluster centers in the embedding space, enforcing cross-embodiment alignment by sharing the same set of prototypes across human and robot data.
• A skill alignment transformer is used during inference to robustly match human video actions to robot visual observations, even under embodiment differences and execution failures.
• The framework employs a skill-conditioned diffusion policy that maps identified human skills to executable robot actions using a conditional score-based generative model.
• During inference, the system detects and aligns skills from a single human prompt video and composes them into a sequence of actions to achieve new, unseen tasks.
• The method uses data augmentation (color jitter, cropping, blur, grayscale) during training to improve robustness and generalization of the learned representations.

Experimental results

Research questions

• RQ1 Can a self-supervised representation learning framework discover shared skill prototypes from unlabeled human and robot manipulation videos across different physical embodiments?
• RQ2 How effectively can a skill-conditioned diffusion policy transfer human-identified skills to a robot’s action space with minimal fine-tuning?
• RQ3 To what extent can the discovered skills be composed to generalize to unseen tasks using only a single human demonstration video?
• RQ4 How does the proposed method compare to baseline imitation learning approaches in terms of zero-shot generalization and robustness under embodiment differences?
• RQ5 Can the framework achieve high performance in both simulated and real-world environments without requiring task labels or expert-annotated trajectories?

Key findings

• XSkill successfully discovers cross-embodiment skill prototypes from unaligned human and robot videos, achieving effective alignment between different physical embodiments through shared prototypes and contrastive learning.
• In simulation, the framework achieved a 92.3% success rate on unseen three-task compositions and 85.6% on four-task compositions, outperforming baseline methods in zero-shot generalization.
• In real-world experiments, XSkill achieved a 78.9% success rate on unseen three-task compositions and 68.4% on four-task compositions, demonstrating robustness to real-world variations and embodiment differences.
• The skill alignment transformer significantly improved skill detection accuracy under visual and action-space discrepancies, reducing misalignment errors by 41% compared to baseline alignment methods.
• The framework generalizes effectively to tasks not seen during training, including novel task sequences and sub-task orders, confirming its compositional generalization capability.
• The ablation study confirmed that both the shared skill prototypes and the diffusion policy are critical components, with performance dropping by over 50% when either is removed.