[Paper Review] ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations

Tongzhou Mu, Zhan Ling+7 more|arXiv (Cornell University)|Jul 30, 2021

3D Shape Modeling and AnalysisEngineering89 references27 citations

TL;DR

ManiSkill introduces a large-scale, open-source benchmark for generalizable 3D-vision–driven manipulation skills across diverse articulated objects, with 36k demonstrations and four tasks in a full-physics simulator.

ABSTRACT

Object manipulation from 3D visual inputs poses many challenges on building generalizable perception and policy models. However, 3D assets in existing benchmarks mostly lack the diversity of 3D shapes that align with real-world intra-class complexity in topology and geometry. Here we propose SAPIEN Manipulation Skill Benchmark (ManiSkill) to benchmark manipulation skills over diverse objects in a full-physics simulator. 3D assets in ManiSkill include large intra-class topological and geometric variations. Tasks are carefully chosen to cover distinct types of manipulation challenges. Latest progress in 3D vision also makes us believe that we should customize the benchmark so that the challenge is inviting to researchers working on 3D deep learning. To this end, we simulate a moving panoramic camera that returns ego-centric point clouds or RGB-D images. In addition, we would like ManiSkill to serve a broad set of researchers interested in manipulation research. Besides supporting the learning of policies from interactions, we also support learning-from-demonstrations (LfD) methods, by providing a large number of high-quality demonstrations (~36,000 successful trajectories, ~1.5M point cloud/RGB-D frames in total). We provide baselines using 3D deep learning and LfD algorithms. All code of our benchmark (simulator, environment, SDK, and baselines) is open-sourced, and a challenge facing interdisciplinary researchers will be held based on the benchmark.

Motivation & Objective

• Motivate and enable evaluation of object-level generalizability in manipulation from 3D visual inputs.
• Provide diverse articulated objects with rich topology/geometry variation to test intra-class generalization.
• Offer multiple task types that cover distinct manipulation challenges (revolute, prismatic, planar, and unconstrained motions).
• Support learning-from-demonstrations (LfD) with a large dataset of successful trajectories to facilitate BC/ offline RL baselines.
• Foster cross-disciplinary research by providing an open, multi-track benchmark (vision, RL, robotics) with scalable data collection.

Proposed method

• Design of ManiSkill with four manipulation tasks: OpenCabinetDoor, OpenCabinetDrawer, PushChair, MoveBucket, each with varied articulated objects.
• Use ego-centric panoramic 3D observations (point clouds, RGB-D) from robot-mounted cameras to enable 3D perception.
• Collect ~36,000 successful demonstrations (~1.5M point cloud/RGB-D frames) via an RL-based, scalable pipeline with a shared reward template and MPC-assisted verification.
• Provide baseline 3D deep learning policies (PointNet; PointNet + Transformer) and learning-from-demonstrations approaches (Imitation Learning BC; Offline RL BCQ, TD3+BC).
• Partition training/test objects within each task to evaluate object-level generalization, under multiple tracks (No Interactions, No External Annotations, No Restrictions).
• Utilize PartNet-Mobility assets with manual post-processing (convex decomposition, artifact removal) and verification to ensure solvable environments.

Experimental results

Research questions

• RQ1 Can policies learn object-level generalizable manipulation skills from 3D visual inputs across diverse intra-class object variations?
• RQ2 How well do 3D deep learning architectures (PointNet, Transformer) and LfD methods perform under object-level generalization when trained on a diverse set of objects?
• RQ3 What is the effect of observation modality (point cloud vs RGB-D) on generalization performance in ManiSkill?
• RQ4 Do offline RL methods outperform behavior cloning when demonstrations are all successful, and under what conditions?
• RQ5 How do different task motions (revolute, prismatic, planar, unconstrained) challenge policy learning and generalization?

Key findings

• Large intra-class variation in topology/geometry supports evaluation of object-level generalizability.
• Even with demonstrations, overall generalization performance remains challenging; training/test performance gaps are evident across tasks.
• PointNet + Transformer with BC offers the best object-level generalization among tested baselines, yet mean test success rates remain modest.
• Offline RL methods (BCQ, TD3+BC) did not consistently outperform BC on the provided demonstrations, highlighting data and task complexity.
• Demonstration count improves performance; more trajectories lead to higher success rates, but generalization remains nontrivial across unseen objects.
• Three-dimensional input strategies (point clouds with segmentation masks) plus robot-state concatenation are important design choices for perception and policy learning.