[Paper Review] ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills

Jiayuan Gu, Fanbo Xiang+13 more|arXiv (Cornell University)|Feb 9, 2023

Robot Manipulation and Learning21 citations

TL;DR

ManiSkill2 offers a unified, fully dynamic 3D manipulation benchmark with 20 task families, 2000+ objects, 4M+ demonstrations, multi-controller support, fast visual RL, and soft-body physics for generalizable manipulation research.

ABSTRACT

Generalizable manipulation skills, which can be composed to tackle long-horizon and complex daily chores, are one of the cornerstones of Embodied AI. However, existing benchmarks, mostly composed of a suite of simulatable environments, are insufficient to push cutting-edge research works because they lack object-level topological and geometric variations, are not based on fully dynamic simulation, or are short of native support for multiple types of manipulation tasks. To this end, we present ManiSkill2, the next generation of the SAPIEN ManiSkill benchmark, to address critical pain points often encountered by researchers when using benchmarks for generalizable manipulation skills. ManiSkill2 includes 20 manipulation task families with 2000+ object models and 4M+ demonstration frames, which cover stationary/mobile-base, single/dual-arm, and rigid/soft-body manipulation tasks with 2D/3D-input data simulated by fully dynamic engines. It defines a unified interface and evaluation protocol to support a wide range of algorithms (e.g., classic sense-plan-act, RL, IL), visual observations (point cloud, RGBD), and controllers (e.g., action type and parameterization). Moreover, it empowers fast visual input learning algorithms so that a CNN-based policy can collect samples at about 2000 FPS with 1 GPU and 16 processes on a regular workstation. It implements a render server infrastructure to allow sharing rendering resources across all environments, thereby significantly reducing memory usage. We open-source all codes of our benchmark (simulator, environments, and baselines) and host an online challenge open to interdisciplinary researchers.

Motivation & Objective

• Promote reproducible research on generalizable manipulation skills with diverse object-level variations and fully dynamic simulation.
• Provide a unified OpenAI Gym interface supporting multiple observation modes and controllers.
• Enable fast data collection for visual reinforcement learning and enable cross-framework comparisons.
• Support soft-body and rigid-body interactions with 2-way coupling between soft-body and rigid-body simulators.
• Facilitate sense-plan-act, imitation learning, and reinforcement learning in a common benchmark with an online challenge.

Proposed method

• Introduce 20 task families spanning stationary/mobile-base, single/dual-arm, and rigid/soft-body manipulation.
• Implement fully dynamic simulation with 2000+ object models and 4M+ demonstrations under a unified Gym interface.
• Develop real-time GPU-accelerated soft-body MPM simulator with 2-way coupling to rigid-body simulators (SAPIEN) for hard-soft interactions.
• Provide multiple controllers (joint-space, delta joint-space, delta end-effector pose, etc.) and a demonstration-action-space conversion to align demonstrations with different controllers.
• Enable fast sampling for visual RL with asynchronous rendering and a render-server to share GPU resources across environments.
• Offer tools for converting demonstration actions across controller spaces to preserve dynamic processes.

Figure 1: ManiSkill2 provides a unified, fast, and accessible system that encompasses well-curated manipulation tasks (e.g., stationary/mobile-base, single/dual-arm, rigid/soft-body).

Experimental results

Research questions

• RQ1 How can a unified benchmark support evaluation of generic and generalizable manipulation skills across diverse task families and object variations?
• RQ2 What are the benefits and challenges of integrating rigid and soft-body dynamics in a single high-fidelity, real-time simulation environment for manipulation?
• RQ3 How do different controllers and action-space representations affect learning, exploration, and sim2real transfer in manipulation tasks?
• RQ4 Can large-scale demonstrations be effectively converted across controllers to enable broad analyses in imitation and reinforcement learning?
• RQ5 What is the performance impact of fast visual RL pipelines (async rendering and render-server) on sample throughput and memory usage?

Key findings

• ManiSkill2 supports 20 task families with 2000+ objects and 4M+ demonstration frames in a unified interface.
• It achieves high-throughput visual RL with asynchronous rendering and a render server, enabling 2000+ FPS under certain settings.
• It introduces real-time soft-body simulation with 2-way coupling to rigid-body simulators, enabling interactions with robots and assets.
• Behavior cloning performs poorly on rigid-body tasks but demonstrations augmented RL (DAPG+PPO) shows improved results on some rigid tasks; high-precision assembly tasks remain challenging.
• Visual RL policies trained in ManiSkill2 show partial sim2real transfer potential, with real-world PickCube achieving 60% success after 10M-time-step simulation training.
• The multi-controller conversion system is crucial; changing controller types significantly impacts success in tasks like PickSingleYCB.

Figure 2: Two pipelines for visual RL sample collection. (a) Sequential pipeline. (b) Our pipeline with asynchronous rendering and render server improves CPU utilization, reduces data transfer, and saves memory.